Nov 03 23:28:12.648013 ip-10-0-138-239 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 23:28:13.054996 ip-10-0-138-239 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:13.054996 ip-10-0-138-239 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 23:28:13.054996 ip-10-0-138-239 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:13.054996 ip-10-0-138-239 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:13.059387 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059308    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.059387 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059386    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.059387 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059390    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059393    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059397    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059399    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059403    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059406    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059409    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059412    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059416    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059421    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059424    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059428    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059431    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059434    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059437    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059440    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059442    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059445    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059448    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.059485 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059450    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059453    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059456    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059459    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059461    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059464    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059467    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059469    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059472    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059475    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059477    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059480    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059482    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059486    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059489    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059492    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059495    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059498    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059500    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.059936 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059503    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059507    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059509    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059511    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059514    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059517    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059520    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059522    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059525    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059527    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059530    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059533    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059536    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059538    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059541    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059544    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059547    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059549    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059552    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059554    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.060376 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059557    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059559    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059562    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059564    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059567    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059569    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059572    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059575    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059578    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059580    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059583    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059586    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059588    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059590    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059593    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059595    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059598    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059601    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059603    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059606    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.060855 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059609    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059611    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059614    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059616    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059619    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059621    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059624    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059627    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059629    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059632    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059634    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059637    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059639    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059642    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059645    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059648    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059652    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059655    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059658    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059661    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.061369 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059664    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059668    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059670    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059673    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059675    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.059678    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060483    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060490    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060492    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060496    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060498    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060501    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060504    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060507    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060510    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060512    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060515    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060518    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060520    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060524    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.062189 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060526    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060529    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060532    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060534    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060537    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060540    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060542    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060545    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060548    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060551    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060553    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060556    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060559    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060562    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060565    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060567    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060570    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060572    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060575    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.062844 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060577    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060579    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060582    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060585    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060589    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060593    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060597    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060600    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060603    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060606    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060610    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060613    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060616    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060618    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060621    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060624    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060626    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060629    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060631    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.063298 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060634    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060636    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060639    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060641    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060644    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060647    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060650    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060652    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060655    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060657    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060660    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060663    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060665    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060668    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060670    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060673    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060676    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060678    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060681    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.063793 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060683    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060686    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060689    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060692    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060694    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060697    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060699    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060702    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060704    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060707    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060710    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060713    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060715    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060718    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060721    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060723    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060728    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060730    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060733    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060750    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.064245 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060753    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060756    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060758    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060761    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060764    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060766    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060769    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060772    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060774    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060778    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060782    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060784    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060787    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060790    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.060794    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062041    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062055    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062060    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062065    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062070    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.064716 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062074    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062079    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062090    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062094    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062098    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062103    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062107    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062111    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062116    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062121    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062125    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062129    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062134    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062138    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062142    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062151    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062155    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062159    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062164    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.065208 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062168    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062173    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062177    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062182    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062186    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062191    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062195    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062201    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062206    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062215    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062219    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062224    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062228    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062232    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062236    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062240    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062245    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062249    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062254    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062258    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.065647 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062262    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062272    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062277    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062281    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062286    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062290    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062294    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062298    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062303    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062307    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062314    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062320    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062326    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062330    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062339    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062343    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062347    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062352    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.066136 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062356    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062360    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062365    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062369    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062373    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062377    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062381    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062385    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062395    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062402    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062407    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062430    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062453    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062481    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062508    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062620    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062624    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062627    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062630    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.066553 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062633    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062635    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062638    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062641    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062643    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062646    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062649    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062651    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062654    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062657    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062660    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062662    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062665    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062668    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062670    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062673    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062675    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062678    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062681    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062683    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.067008 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062686    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062688    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062691    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062693    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.062696    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064597    2580 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064609    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064620    2580 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064625    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064630    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064633    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064638    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064642    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064645    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064649    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064652    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064656    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064659    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064662    2580 flags.go:64] FLAG: --cgroup-root=""
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064665    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064668    2580 flags.go:64] FLAG: --client-ca-file=""
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064671    2580 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064674    2580 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064679    2580 flags.go:64] FLAG: --cluster-domain=""
Nov 03 23:28:13.067468 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064682    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064685    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064688    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064692    2580 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064696    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064699    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064703    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064707    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064711    2580 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064714    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064717    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064720    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064723    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064728    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064731    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064735    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064748    2580 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064751    2580 flags.go:64] FLAG: --enable-server="true"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064756    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064762    2580 flags.go:64] FLAG: --event-burst="100"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064765    2580 flags.go:64] FLAG: --event-qps="50"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064768    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064771    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064774    2580 flags.go:64] FLAG: --eviction-hard=""
Nov 03 23:28:13.068053 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064779    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064782    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064785    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064788    2580 flags.go:64] FLAG: --eviction-soft=""
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064792    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064794    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064798    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064801    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064805    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064808    2580 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064812    2580 flags.go:64] FLAG: --feature-gates=""
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064816    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064819    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064823    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064826    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064830    2580 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064833    2580 flags.go:64] FLAG: --help="false"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064836    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064839    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064842    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064845    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064849    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064852    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 23:28:13.068671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064855    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064858    2580 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064861    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064865    2580 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064868    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064872    2580 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064875    2580 flags.go:64] FLAG: --kube-reserved=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064878    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064882    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064885    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064889    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064892    2580 flags.go:64] FLAG: --lock-file=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064895    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064898    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064901    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064907    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064910    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064913    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064916    2580 flags.go:64] FLAG: --logging-format="text"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064919    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064923    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064926    2580 flags.go:64] FLAG: --manifest-url=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064929    2580 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064934    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064937    2580 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 23:28:13.069269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064941    2580 flags.go:64] FLAG: --max-pods="110"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064944    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064947    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064950    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064953    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064956    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064959    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064962    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064970    2580 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064973    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064978    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064981    2580 flags.go:64] FLAG: --pod-cidr=""
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064984    2580 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064987    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064990    2580 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064993    2580 flags.go:64] FLAG: --port="10250"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064996    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.064999    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0526ede6620beaf57"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065003    2580 flags.go:64] FLAG: --qos-reserved=""
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065006    2580 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065009    2580 flags.go:64] FLAG: --register-node="true"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065012    2580 flags.go:64] FLAG: --register-with-taints=""
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065016    2580 flags.go:64] FLAG: --registry-burst="10"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065019    2580 flags.go:64] FLAG: --registry-qps="5"
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065022    2580 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065025    2580 flags.go:64] FLAG: --reserved-memory=""
Nov 03 23:28:13.069868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065029    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065033    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065036    2580 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065039    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065042    2580 flags.go:64] FLAG: --runonce="false"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065044    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065047    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065050    2580 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065053    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065056    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065060    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065063    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065066    2580 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065069    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065071    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065074    2580 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065078    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065082    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065085    2580 flags.go:64] FLAG: --system-cgroups=""
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065088    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065095    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065098    2580 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065101    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065105    2580 flags.go:64] FLAG: --tls-min-version=""
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065108    2580 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 23:28:13.070555 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065112    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065115    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065118    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065121    2580 flags.go:64] FLAG: --v="2"
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065125    2580 flags.go:64] FLAG: --version="false"
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065130    2580 flags.go:64] FLAG: --vmodule=""
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065134    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065137    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065278    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065282    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065285    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065287    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065290    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065293    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065295    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065298    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065301    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065303    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065306    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065309    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065311    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.071173 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065314    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065317    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065320    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065323    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065326    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065329    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065331    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065334    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065338    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065340    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065343    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065345    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065349    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065351    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065354    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065356    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065359    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065361    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065364    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065366    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.072079 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065369    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065371    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065374    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065377    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065379    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065382    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065384    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065387    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065389    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065392    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065395    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065397    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065399    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065402    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065405    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065408    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065410    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065413    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065416    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.072926 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065418    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065423    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065426    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065428    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065431    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065433    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065437    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065439    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065442    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065444    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065447    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065450    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065452    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065455    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065457    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065460    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065462    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065465    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065468    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065470    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.073812 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065473    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065475    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065478    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065481    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065483    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065486    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065488    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065491    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065494    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065496    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065499    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065501    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065503    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065508    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065511    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065513    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065516    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065518    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065521    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065524    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.074680 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065527    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065529    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065532    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065536    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065540    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065543    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065547    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065550    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065552    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065555    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065559    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065563    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065566    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065569    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.065595    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065631    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.075558 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065636    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065639    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065642    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065644    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065647    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065650    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065652    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065655    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065658    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065661    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065665    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065668    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065670    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065673    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065676    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065678    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065681    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065684    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065687    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065690    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.076110 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065693    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065696    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065698    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065701    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065704    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065706    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065709    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065711    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065714    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065716    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065719    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065721    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065724    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065730    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065734    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065749    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065752    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065755    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065758    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.076711 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065761    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065765    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065769    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065773    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065776    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065779    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065782    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065784    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065787    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065789    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065792    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065795    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065798    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065800    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065803    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065805    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065808    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065810    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065814    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.077617 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065816    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065818    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065821    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065824    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065826    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065829    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065832    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065836    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065838    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065841    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065844    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065846    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065849    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065851    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065854    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065856    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065860    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065863    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065865    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065868    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.078250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065871    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065873    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065875    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065878    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065881    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065884    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065886    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065889    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065892    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065894    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065897    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065899    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065902    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065904    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065908    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065910    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065920    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065923    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065925    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.078907 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065929    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065932    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065935    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065938    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065940    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065943    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065945    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.065948    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.073678    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.073701    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.073725    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.073732    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073833    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073842    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073848    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073853    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073858    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073864    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073869    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.079363 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073874    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073879    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073883    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073887    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073892    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073896    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073901    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073906    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073911    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073923    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073927    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073931    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073935    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073939    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073944    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073951    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073957    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073961    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073966    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.079899 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073971    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073975    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073979    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073986    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073991    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.073995    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074000    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074004    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074009    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074013    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074017    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074021    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074026    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074031    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074035    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074039    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074043    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074048    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074052    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074056    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.080419 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074060    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074065    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074069    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074073    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074078    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074082    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074086    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074090    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074095    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074099    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074103    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074107    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074111    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074116    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074120    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074124    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074129    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074133    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074138    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074142    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.080915 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074146    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074150    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074154    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074158    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074163    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074168    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074172    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074176    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074180    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074185    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074189    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074192    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074196    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074201    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074205    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074209    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074213    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074218    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074223    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074227    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.081449 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074232    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074236    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074240    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074245    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074249    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074253    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074257    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074262    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074266    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074271    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074275    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074279    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074283    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074288    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074292    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074296    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074301    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074305    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074311    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.082140 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074315    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.074358    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074591    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074599    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074604    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074609    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074613    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074618    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074622    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074626    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074630    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074634    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074638    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074643    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074647    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074651    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.082646 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074655    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074660    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074664    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074668    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074673    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074677    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074681    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074685    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074689    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074694    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074697    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074701    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074705    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074710    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074714    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074718    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074724    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074729    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074733    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.083300 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074752    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074758    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074762    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074766    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074771    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074775    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074779    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074784    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074788    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074792    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074796    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074801    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074805    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074809    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074814    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074818    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074823    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074828    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074832    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074836    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.084095 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074841    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074845    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074849    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074853    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074858    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074862    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074866    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074870    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074877    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074882    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074889    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074894    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074899    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074906    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074912    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074919    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074924    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074928    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074932    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.084794 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074937    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074941    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074945    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074950    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074954    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074958    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074963    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074967    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074972    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074976    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074981    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074985    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074990    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074994    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.074998    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075002    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075006    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075010    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075014    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.085240 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075019    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075023    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075027    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075031    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075035    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075041    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075045    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075051    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075056    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075060    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075064    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075068    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075072    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075077    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075081    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.075124    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:13.085778 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075178    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075185    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075190    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075194    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075200    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075204    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075209    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075214    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075218    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075223    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075227    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075231    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075235    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075239    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075244    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075248    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075252    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075257    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075261    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:13.086181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075265    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075269    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075273    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075279    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075286    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075290    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075294    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075298    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075306    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075311    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075315    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075319    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075323    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075328    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075332    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075337    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075341    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075345    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075349    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075353    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:13.086639 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075357    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075362    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075366    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075370    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075374    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075379    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075383    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075390    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075396    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075400    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075405    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075409    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075414    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075421    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075427    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075431    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075438    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075442    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075447    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:13.087116 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075451    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075457    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075461    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075466    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075470    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075474    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075478    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075482    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075487    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075491    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075495    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075499    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075503    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075507    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075511    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075516    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075520    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075524    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075528    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:13.087581 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075533    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075537    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075541    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075545    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075549    2580 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075553    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075558    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075562    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075566    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075570    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075576    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075580    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075585    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075589    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075595    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075602    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075606    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075610    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075615    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075619    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:13.088119 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075623    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075627    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075631    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075636    2580 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075640    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075644    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075648    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075652    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:13.075657    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.075755    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.078478    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.079607    2580 server.go:1014] "Starting client certificate rotation"
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.079715    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.080416    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.083006    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:28:13.088653 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.088269    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:28:13.096971 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.096942    2580 log.go:25] "Validated CRI v1 runtime API"
Nov 03 23:28:13.099472 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.099454    2580 log.go:25] "Validated CRI v1 image API"
Nov 03 23:28:13.101533 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.101503    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 23:28:13.104944 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.104921    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ba5eac8c-fe68-40b6-84d6-fbad63022036:/dev/nvme0n1p3 bbb731a6-bffc-4465-9458-b86bb2c92502:/dev/nvme0n1p4]
Nov 03 23:28:13.105007 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.104942    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 23:28:13.110347 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.110233    2580 manager.go:217] Machine: {Timestamp:2026-11-03 23:28:13.109036434 +0000 UTC m=+0.341959120 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099615 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ef07376810e8ec536a66fc98485a SystemUUID:ec21ef07-3768-10e8-ec53-6a66fc98485a BootID:814e9d61-da51-4ad3-bee6-346d5921b62b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:c2:c2:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:c2:c2:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:8e:30:ff:a3:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 23:28:13.110876 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.110866    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 23:28:13.110967 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.110956    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 23:28:13.111752 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.111723    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 23:28:13.114060 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114035    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 23:28:13.114198 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114063    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-239.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 23:28:13.114242 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114200    2580 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 23:28:13.114242 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114208    2580 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 23:28:13.114242 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114218    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:28:13.114242 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114232    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:28:13.114974 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.114964    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 23:28:13.118504 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118491    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 23:28:13.118625 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118615    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 23:28:13.118668 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118647    2580 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 23:28:13.118668 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118658    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 23:28:13.118729 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118677    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 23:28:13.118729 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118687    2580 kubelet.go:405] "Adding apiserver pod source"
Nov 03 23:28:13.118729 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.118695    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 23:28:13.119066 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.119054    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:28:13.121650 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.121632    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 23:28:13.127158 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.127145    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:28:13.127903 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.127892    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 23:28:13.127953 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.127921    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 23:28:13.128077 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128065    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 23:28:13.128128 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128086    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 23:28:13.128128 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128095    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 23:28:13.128128 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128103    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 23:28:13.128128 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128110    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 23:28:13.128128 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128118    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 23:28:13.128570 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128561    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 23:28:13.128598 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128570    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 23:28:13.128598 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128580    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 23:28:13.128598 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128586    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 23:28:13.128733 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128618    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 23:28:13.128733 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.128628    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 23:28:13.129310 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.129300    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 23:28:13.129310 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.129310    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 23:28:13.133223 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.133209    2580 server.go:1263] "Started kubelet"
Nov 03 23:28:13.133363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.133289    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 23:28:13.133420 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.133307    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 23:28:13.133420 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.133412    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 23:28:13.134136 ip-10-0-138-239 systemd[1]: Started Kubernetes Kubelet.
Nov 03 23:28:13.134486 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.134352    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-239.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 23:28:13.134905 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.134852    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 23:28:13.142042 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.141263    2580 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 23:28:13.142042 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.141909    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 23:28:13.144237 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.144218    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 23:28:13.144722 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.144708    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 23:28:13.145001 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.144965    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.145398 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145378    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 23:28:13.145398 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145396    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 23:28:13.145546 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145377    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 23:28:13.145546 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145530    2580 factory.go:55] Registering systemd factory
Nov 03 23:28:13.145546 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145545    2580 factory.go:223] Registration of the systemd container factory successfully
Nov 03 23:28:13.145732 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145713    2580 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 23:28:13.145732 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145730    2580 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 23:28:13.145866 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145807    2580 factory.go:153] Registering CRI-O factory
Nov 03 23:28:13.145866 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145819    2580 factory.go:223] Registration of the crio container factory successfully
Nov 03 23:28:13.145935 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145872    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 23:28:13.145935 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145891    2580 factory.go:103] Registering Raw factory
Nov 03 23:28:13.145935 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.145902    2580 manager.go:1199] Started watching for new ooms in manager
Nov 03 23:28:13.146776 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.146765    2580 manager.go:319] Starting recovery of all containers
Nov 03 23:28:13.153555 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.152564    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-239.us-east-2.compute.internal.18e4acc0359878ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-239.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-239.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:13.133183147 +0000 UTC m=+0.366105829,LastTimestamp:2026-11-03 23:28:13.133183147 +0000 UTC m=+0.366105829,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-239.us-east-2.compute.internal,}"
Nov 03 23:28:13.153709 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.153670    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-239.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 23:28:13.156695 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.156640    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-239.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 23:28:13.156817 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.156798    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 23:28:13.158032 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.158018    2580 manager.go:324] Recovery completed
Nov 03 23:28:13.160797 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.160771    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 23:28:13.163233 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.163220    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.165153 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165125    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.165219 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165171    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.165219 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165186    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.165794 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165769    2580 cpu_manager.go:226] "Starting" policy="none"
Nov 03 23:28:13.165794 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165786    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.165794 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165793    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 23:28:13.165953 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.165812    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 23:28:13.166522 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.166509    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.166574 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.166534    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.166574 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.166544    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.168023 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.168013    2580 policy_none.go:50] "Start"
Nov 03 23:28:13.168063 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.168028    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 23:28:13.168063 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.168050    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 23:28:13.168560 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.168468    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-239.us-east-2.compute.internal.18e4acc037804ad6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-239.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-239.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-239.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:13.165152982 +0000 UTC m=+0.398075673,LastTimestamp:2026-11-03 23:28:13.165152982 +0000 UTC m=+0.398075673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-239.us-east-2.compute.internal,}"
Nov 03 23:28:13.169720 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.169706    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b2m4j"
Nov 03 23:28:13.170113 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.170102    2580 policy_none.go:44] "Start"
Nov 03 23:28:13.179281 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.179203    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-239.us-east-2.compute.internal.18e4acc03780ae6f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-239.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-239.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-239.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:13.165178479 +0000 UTC m=+0.398101165,LastTimestamp:2026-11-03 23:28:13.165178479 +0000 UTC m=+0.398101165,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-239.us-east-2.compute.internal,}"
Nov 03 23:28:13.181659 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.181642    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b2m4j"
Nov 03 23:28:13.183151 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.183087    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-239.us-east-2.compute.internal.18e4acc03780e504  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-239.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-239.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-239.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:13.165192452 +0000 UTC m=+0.398115141,LastTimestamp:2026-11-03 23:28:13.165192452 +0000 UTC m=+0.398115141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-239.us-east-2.compute.internal,}"
Nov 03 23:28:13.203352 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.203315    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 23:28:13.207277 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207265    2580 manager.go:341] "Starting Device Plugin manager"
Nov 03 23:28:13.207363 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.207299    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 23:28:13.207363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207307    2580 server.go:87] "Starting device plugin registration server"
Nov 03 23:28:13.207585 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207573    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 23:28:13.207618 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207588    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 23:28:13.207856 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207666    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 23:28:13.207856 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207769    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 23:28:13.207856 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.207778    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 23:28:13.208284 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.208267    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 23:28:13.208352 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.208307    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.308232 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.308150    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.309267 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.309242    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.309368 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.309285    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.309368 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.309301    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.309368 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.309340    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.333222 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.319694    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.333222 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.319856    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-239.us-east-2.compute.internal\": node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.358158 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.358138    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.458830 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.458777    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.523283 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.523242    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 23:28:13.524512 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.524492    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 23:28:13.524561 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.524523    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 23:28:13.524561 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.524549    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 23:28:13.525430 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.524589    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 23:28:13.532477 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.532456    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:13.559828 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.559767    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.626260 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.626228    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"]
Nov 03 23:28:13.626354 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.626303    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.627318 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.627297    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.627389 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.627334    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.627389 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.627345    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.629581 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.629569    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.629763 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.629731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.629802 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.629780    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.630335 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.630320    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.630399 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.630341    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.630399 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.630349    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.630399 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.630359    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.630399 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.630362    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.630399 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.630372    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.632402 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.632383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.632481 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.632412    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:13.633087 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.633072    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:13.633199 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.633101    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:13.633199 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.633117    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:13.647307 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.647284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.647381 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.647311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14f459d1824c1ea630aa90a99306c647-config\") pod \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"14f459d1824c1ea630aa90a99306c647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.647381 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.647338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.660468 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.660444    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.663353 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.663334    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.665168 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.665153    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.748153 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.748123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.748153 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.748155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14f459d1824c1ea630aa90a99306c647-config\") pod \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"14f459d1824c1ea630aa90a99306c647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.748310 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.748175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.748310 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.748237    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.748310 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.748240    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14f459d1824c1ea630aa90a99306c647-config\") pod \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"14f459d1824c1ea630aa90a99306c647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.748310 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.748240    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.761248 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.761225    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.861830 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.861762    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.962224 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:13.962193    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:13.967347 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.967331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:13.968434 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:13.968419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:14.063152 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:14.063123    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:14.079855 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.079834    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 23:28:14.079969 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.079949    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 23:28:14.164232 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:14.164175    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:14.185922 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.185887    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 23:23:13 +0000 UTC" deadline="2028-04-03 15:53:43.637128321 +0000 UTC"
Nov 03 23:28:14.185922 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.185921    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12400h25m29.4512121s"
Nov 03 23:28:14.207447 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.207423    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:14.245255 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.245227    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:14.260330 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.260312    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:14.262266 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.262254    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:14.269107 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.269088    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:14.392960 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.392937    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:14.660749 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:14.660704    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f459d1824c1ea630aa90a99306c647.slice/crio-78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170 WatchSource:0}: Error finding container 78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170: Status 404 returned error can't find the container with id 78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170
Nov 03 23:28:14.665078 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:14.665055    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a45f1006b1620e684bf8ebed402834.slice/crio-eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63 WatchSource:0}: Error finding container eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63: Status 404 returned error can't find the container with id eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63
Nov 03 23:28:14.666256 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.666238    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:28:14.712619 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:14.712584    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:15.064262 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.064228    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:15.121567 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.121540    2580 apiserver.go:52] "Watching apiserver"
Nov 03 23:28:15.131380 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.131342    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 23:28:15.131785 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.131754    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6n2fd","openshift-cluster-node-tuning-operator/tuned-dxj77","openshift-dns/node-resolver-c99qk","openshift-network-diagnostics/network-check-target-7cjjg","openshift-ovn-kubernetes/ovnkube-node-ncc9n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal","openshift-multus/multus-jt2sr","openshift-network-operator/iptables-alerter-6czcz","kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4","kube-system/global-pull-secret-syncer-pcvlc","kube-system/konnectivity-agent-b9jnr","openshift-image-registry/node-ca-cdxlq","openshift-multus/multus-additional-cni-plugins-cqzg8"]
Nov 03 23:28:15.134517 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.134492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.136801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.136778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.138454 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138434    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.138454 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138447    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 23:28:15.138588 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138527    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 23:28:15.138588 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138558    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 23:28:15.138704 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138683    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.138797 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138780    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w5s9r\""
Nov 03 23:28:15.138975 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.138948    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 23:28:15.139378 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.139360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.139573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.139558    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.139650 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.139614    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.139824 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.139810    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 23:28:15.139921 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.139902    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgkz4\""
Nov 03 23:28:15.141641 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.141622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:15.141764 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.141712    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:15.142280 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.142253    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.142367 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.142313    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.142639 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.142381    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q987z\""
Nov 03 23:28:15.144069 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.144053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.145048 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.145028    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 23:28:15.146448 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.146430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.146543 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.146508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:15.148831 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.148801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.149134 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.148990    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lsnsx\""
Nov 03 23:28:15.151745 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.151725    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.154476 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.154456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.154580 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.154534    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:15.157069 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-kubelet\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovn-node-metrics-cert\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrrf\" (UniqueName: \"kubernetes.io/projected/a1e7a5aa-87e3-4afd-ae59-801a559622f6-kube-api-access-gbrrf\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.157192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-systemd-units\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157371 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-netns\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157371 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157292    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.157371 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd9bbc11-d972-4662-bb0d-b1d8744673c9-host\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.157518 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-kubelet\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.157518 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.157518 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-ovn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157518 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157723 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4738a53-824f-4a8b-ab86-b160b9f14c0e-hosts-file\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.157723 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4738a53-824f-4a8b-ab86-b160b9f14c0e-tmp-dir\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.157723 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-socket-dir-parent\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.157723 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-etc-kubernetes\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.157723 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-etc-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.157723 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-log-socket\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158106 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158106 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:15.158106 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.157982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158106 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-conf-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-daemon-config\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-systemd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-var-lib-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-script-lib\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjr2\" (UniqueName: \"kubernetes.io/projected/c4738a53-824f-4a8b-ab86-b160b9f14c0e-kube-api-access-vdjr2\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cni-binary-copy\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-netns\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62llt\" (UniqueName: \"kubernetes.io/projected/1244c649-9fe7-40c5-9b55-443a2529c710-kube-api-access-62llt\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-bin\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158413 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-netd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.158951 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98v7q\" (UniqueName: \"kubernetes.io/projected/cd9bbc11-d972-4662-bb0d-b1d8744673c9-kube-api-access-98v7q\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.158951 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-system-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158951 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-bin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.158951 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-multus\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.159130 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.158962    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 23:28:15.159201 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-hostroot\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.159781 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.159883 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-env-overrides\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.159883 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpvn\" (UniqueName: \"kubernetes.io/projected/5bc8021d-2812-40f5-99ba-85e10f04ef4f-kube-api-access-bvpvn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.159883 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd9bbc11-d972-4662-bb0d-b1d8744673c9-serviceca\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.160035 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cnibin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.160035 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-k8s-cni-cncf-io\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.160035 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-multus-certs\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.160035 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.159994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-slash\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.160239 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.160054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-node-log\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.160239 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.160096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-config\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.160755 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.160720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.164648 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.164626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.244669 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.244633    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rh4ps"
Nov 03 23:28:15.246678 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.246659    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 23:28:15.248246 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248221    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.248369 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248345    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 23:28:15.248444 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248379    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.248444 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248403    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 23:28:15.248530 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248453    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.248530 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248486    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.248530 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248496    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b74w2\""
Nov 03 23:28:15.248530 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248498    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 23:28:15.248530 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248511    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248557    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248571    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248614    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9hzk\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248626    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248633    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jctss\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248635    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248656    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 23:28:15.248692 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248661    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.249088 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248710    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dd24d\""
Nov 03 23:28:15.249088 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248832    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwn6\""
Nov 03 23:28:15.249088 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248896    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 23:28:15.249088 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.248954    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:15.257915 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.257516    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rh4ps"
Nov 03 23:28:15.260505 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smznp\" (UniqueName: \"kubernetes.io/projected/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-kube-api-access-smznp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.260638 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.260638 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.260638 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4738a53-824f-4a8b-ab86-b160b9f14c0e-tmp-dir\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.260638 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-etc-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.260893 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.260893 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.260893 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-conf-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.260893 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcd47e4a-ca1b-4557-82d0-200bdb384976-iptables-alerter-script\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.260893 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcr4z\" (UniqueName: \"kubernetes.io/projected/fcd47e4a-ca1b-4557-82d0-200bdb384976-kube-api-access-rcr4z\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.260893 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysconfig\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-tuned\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-script-lib\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.260986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjr2\" (UniqueName: \"kubernetes.io/projected/c4738a53-824f-4a8b-ab86-b160b9f14c0e-kube-api-access-vdjr2\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-registration-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-run\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-netd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-system-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.261168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-bin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-multus\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cnibin\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-env-overrides\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpvn\" (UniqueName: \"kubernetes.io/projected/5bc8021d-2812-40f5-99ba-85e10f04ef4f-kube-api-access-bvpvn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-modprobe-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-systemd\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47845d6a-a330-49f4-be05-d3a8aeb677dc-agent-certs\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47845d6a-a330-49f4-be05-d3a8aeb677dc-konnectivity-ca\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.261501 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-kubernetes\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-var-lib-kubelet\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-systemd-units\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-netns\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd9bbc11-d972-4662-bb0d-b1d8744673c9-host\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-kubelet\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd47e4a-ca1b-4557-82d0-200bdb384976-host-slash\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-ovn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.261926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4738a53-824f-4a8b-ab86-b160b9f14c0e-hosts-file\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-socket-dir-parent\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.261977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-etc-kubernetes\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-sys-fs\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-conf\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-host\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-log-socket\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-daemon-config\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62llt\" (UniqueName: \"kubernetes.io/projected/1244c649-9fe7-40c5-9b55-443a2529c710-kube-api-access-62llt\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-socket-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcl2v\" (UniqueName: \"kubernetes.io/projected/63593602-b03b-47c5-9d74-a95ff2abfddb-kube-api-access-jcl2v\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.262289 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-systemd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-var-lib-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cni-binary-copy\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-netns\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scd5g\" (UniqueName: \"kubernetes.io/projected/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-kube-api-access-scd5g\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-bin\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98v7q\" (UniqueName: \"kubernetes.io/projected/cd9bbc11-d972-4662-bb0d-b1d8744673c9-kube-api-access-98v7q\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-hostroot\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-kubelet-config\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-dbus\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-sys\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd9bbc11-d972-4662-bb0d-b1d8744673c9-serviceca\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cnibin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-k8s-cni-cncf-io\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-multus-certs\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.262770 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-tmp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.262775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-slash\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-node-log\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-config\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-kubelet\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovn-node-metrics-cert\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrrf\" (UniqueName: \"kubernetes.io/projected/a1e7a5aa-87e3-4afd-ae59-801a559622f6-kube-api-access-gbrrf\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-lib-modules\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-device-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.263504    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:15.263586 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.263577    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:15.76354854 +0000 UTC m=+2.996471226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:15.264150 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263788    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-ovn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.264150 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263863    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-bin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.264150 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.263905    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.264304 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.264166    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4738a53-824f-4a8b-ab86-b160b9f14c0e-tmp-dir\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.264304 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.264215    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-etc-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.264462 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.264335    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.264462 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.264361    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-conf-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.264914 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.264892    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-script-lib\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.265253 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.265219    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-netd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.265323 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.265296    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-system-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.265379 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.265338    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-multus\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.265715 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.265689    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-env-overrides\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.265990 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.265972    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-systemd-units\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.266064 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266017    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-netns\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.266064 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd9bbc11-d972-4662-bb0d-b1d8744673c9-host\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.266172 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266088    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-kubelet\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.266172 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266125    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.266286 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266171    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4738a53-824f-4a8b-ab86-b160b9f14c0e-hosts-file\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.266286 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266215    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-socket-dir-parent\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.266286 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266253    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-etc-kubernetes\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.266439 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.266292    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-log-socket\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.267032 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.267012    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-daemon-config\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.267278 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.267254    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-systemd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.267345 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.267304    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-var-lib-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.267759 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.267739    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cni-binary-copy\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.267843 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.267792    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-netns\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.269269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.268978    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.269269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.268998    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cnibin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.269269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.269010    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-k8s-cni-cncf-io\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.269670    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-multus-certs\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.269683    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-bin\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.269740    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-slash\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.269804    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-hostroot\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.270055    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.270090    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-node-log\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.270177    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd9bbc11-d972-4662-bb0d-b1d8744673c9-serviceca\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.270280    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-kubelet\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.270842 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.270416    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-config\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.274355 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.274334    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovn-node-metrics-cert\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.278202 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.278183    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:15.278300 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.278206    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:15.278300 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.278220    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8662p for pod openshift-network-diagnostics/network-check-target-7cjjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:15.278300 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.278286    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p podName:d2d422d3-fe51-4985-8577-4ddfcc24ba8c nodeName:}" failed. No retries permitted until 2026-11-03 23:28:15.778266955 +0000 UTC m=+3.011189644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8662p" (UniqueName: "kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p") pod "network-check-target-7cjjg" (UID: "d2d422d3-fe51-4985-8577-4ddfcc24ba8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:15.280372 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.280314    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjr2\" (UniqueName: \"kubernetes.io/projected/c4738a53-824f-4a8b-ab86-b160b9f14c0e-kube-api-access-vdjr2\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.281710 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.281684    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpvn\" (UniqueName: \"kubernetes.io/projected/5bc8021d-2812-40f5-99ba-85e10f04ef4f-kube-api-access-bvpvn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.283098 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.283069    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62llt\" (UniqueName: \"kubernetes.io/projected/1244c649-9fe7-40c5-9b55-443a2529c710-kube-api-access-62llt\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.285018 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.284705    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrrf\" (UniqueName: \"kubernetes.io/projected/a1e7a5aa-87e3-4afd-ae59-801a559622f6-kube-api-access-gbrrf\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.285451 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.285426    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98v7q\" (UniqueName: \"kubernetes.io/projected/cd9bbc11-d972-4662-bb0d-b1d8744673c9-kube-api-access-98v7q\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.363973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcd47e4a-ca1b-4557-82d0-200bdb384976-iptables-alerter-script\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr4z\" (UniqueName: \"kubernetes.io/projected/fcd47e4a-ca1b-4557-82d0-200bdb384976-kube-api-access-rcr4z\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysconfig\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-tuned\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-registration-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-run\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cnibin\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-modprobe-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-systemd\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47845d6a-a330-49f4-be05-d3a8aeb677dc-agent-certs\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47845d6a-a330-49f4-be05-d3a8aeb677dc-konnectivity-ca\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-kubernetes\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-var-lib-kubelet\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd47e4a-ca1b-4557-82d0-200bdb384976-host-slash\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.365584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-sys-fs\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-conf\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-host\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-socket-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcl2v\" (UniqueName: \"kubernetes.io/projected/63593602-b03b-47c5-9d74-a95ff2abfddb-kube-api-access-jcl2v\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scd5g\" (UniqueName: \"kubernetes.io/projected/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-kube-api-access-scd5g\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-kubelet-config\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-dbus\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-sys\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-tmp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-lib-modules\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-device-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.366613 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smznp\" (UniqueName: \"kubernetes.io/projected/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-kube-api-access-smznp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364763    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcd47e4a-ca1b-4557-82d0-200bdb384976-iptables-alerter-script\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364833    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-sys-fs\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.364978    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-conf\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365055    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-socket-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365118    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-device-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365133    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365140    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-systemd\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365180    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365201    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-registration-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365249    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-run\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365282    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-kubernetes\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365307    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-lib-modules\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365339    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-host\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365359    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.367573 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365429    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-kubelet-config\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365454    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-dbus\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365558    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365655    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365700    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cnibin\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365746    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd47e4a-ca1b-4557-82d0-200bdb384976-host-slash\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365797    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-var-lib-kubelet\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365878    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365892    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-modprobe-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365918    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47845d6a-a330-49f4-be05-d3a8aeb677dc-konnectivity-ca\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.365957    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-sys\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.365977    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.366034    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret podName:3e73b381-beec-43fe-ba3d-460efc605f8b nodeName:}" failed. No retries permitted until 2026-11-03 23:28:15.866015285 +0000 UTC m=+3.098937960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret") pod "global-pull-secret-syncer-pcvlc" (UID: "3e73b381-beec-43fe-ba3d-460efc605f8b") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.366080    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysconfig\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.367036    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.368400 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.367068    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.369645 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.369589    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47845d6a-a330-49f4-be05-d3a8aeb677dc-agent-certs\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.370139 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.370109    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-tuned\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.371906 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.371857    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-tmp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.384258 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.384216    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smznp\" (UniqueName: \"kubernetes.io/projected/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-kube-api-access-smznp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.384912 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.384804    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr4z\" (UniqueName: \"kubernetes.io/projected/fcd47e4a-ca1b-4557-82d0-200bdb384976-kube-api-access-rcr4z\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.385469 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.385445    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcl2v\" (UniqueName: \"kubernetes.io/projected/63593602-b03b-47c5-9d74-a95ff2abfddb-kube-api-access-jcl2v\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.387863 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.387842    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scd5g\" (UniqueName: \"kubernetes.io/projected/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-kube-api-access-scd5g\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.466119 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.466084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:15.468918 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.468863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:15.474780 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.474600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:15.475719 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.475672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:15.477372 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.477343    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9bbc11_d972_4662_bb0d_b1d8744673c9.slice/crio-25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097 WatchSource:0}: Error finding container 25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097: Status 404 returned error can't find the container with id 25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097
Nov 03 23:28:15.482884 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.482855    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4738a53_824f_4a8b_ab86_b160b9f14c0e.slice/crio-43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9 WatchSource:0}: Error finding container 43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9: Status 404 returned error can't find the container with id 43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9
Nov 03 23:28:15.483610 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.483382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:15.488858 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.488831    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc8021d_2812_40f5_99ba_85e10f04ef4f.slice/crio-b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b WatchSource:0}: Error finding container b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b: Status 404 returned error can't find the container with id b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b
Nov 03 23:28:15.489879 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.489843    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e7a5aa_87e3_4afd_ae59_801a559622f6.slice/crio-442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e WatchSource:0}: Error finding container 442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e: Status 404 returned error can't find the container with id 442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e
Nov 03 23:28:15.495982 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.495952    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd47e4a_ca1b_4557_82d0_200bdb384976.slice/crio-30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435 WatchSource:0}: Error finding container 30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435: Status 404 returned error can't find the container with id 30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435
Nov 03 23:28:15.499054 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.498593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:15.500968 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.500920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:15.510375 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.510229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:15.511233 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.511203    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63593602_b03b_47c5_9d74_a95ff2abfddb.slice/crio-4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db WatchSource:0}: Error finding container 4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db: Status 404 returned error can't find the container with id 4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db
Nov 03 23:28:15.516108 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.516074    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47845d6a_a330_49f4_be05_d3a8aeb677dc.slice/crio-e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f WatchSource:0}: Error finding container e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f: Status 404 returned error can't find the container with id e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f
Nov 03 23:28:15.531984 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.531812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:15.532192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.532154    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxj77" event={"ID":"3a0dae53-7734-414d-94ca-7bfd0bbc5d62","Type":"ContainerStarted","Data":"458fbe8e1f9b05a9f60ec59dbd96972cfe73164c941195d77066f363db65a0ee"}
Nov 03 23:28:15.534401 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.534348    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b9jnr" event={"ID":"47845d6a-a330-49f4-be05-d3a8aeb677dc","Type":"ContainerStarted","Data":"e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f"}
Nov 03 23:28:15.535916 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.535890    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt2sr" event={"ID":"a1e7a5aa-87e3-4afd-ae59-801a559622f6","Type":"ContainerStarted","Data":"442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e"}
Nov 03 23:28:15.537731 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.537708    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b"}
Nov 03 23:28:15.539298 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.539270    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c99qk" event={"ID":"c4738a53-824f-4a8b-ab86-b160b9f14c0e","Type":"ContainerStarted","Data":"43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9"}
Nov 03 23:28:15.540955 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.540888    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cdxlq" event={"ID":"cd9bbc11-d972-4662-bb0d-b1d8744673c9","Type":"ContainerStarted","Data":"25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097"}
Nov 03 23:28:15.542349 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.542264    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4" event={"ID":"63593602-b03b-47c5-9d74-a95ff2abfddb","Type":"ContainerStarted","Data":"4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db"}
Nov 03 23:28:15.544181 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:15.544151    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bbd0ed_0bf6_401c_8ea6_d759c3909adc.slice/crio-227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278 WatchSource:0}: Error finding container 227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278: Status 404 returned error can't find the container with id 227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278
Nov 03 23:28:15.544181 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.544173    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6czcz" event={"ID":"fcd47e4a-ca1b-4557-82d0-200bdb384976","Type":"ContainerStarted","Data":"30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435"}
Nov 03 23:28:15.547050 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.546999    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"a9a45f1006b1620e684bf8ebed402834","Type":"ContainerStarted","Data":"eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63"}
Nov 03 23:28:15.548777 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.548629    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"14f459d1824c1ea630aa90a99306c647","Type":"ContainerStarted","Data":"78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170"}
Nov 03 23:28:15.768577 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.768510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:15.768727 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.768644    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:15.768727 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.768724    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:16.768703455 +0000 UTC m=+4.001626128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:15.870420 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.870387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:15.870420 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:15.870428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:15.870652 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.870556    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:15.870652 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.870589    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:15.870652 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.870610    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:15.870652 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.870623    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8662p for pod openshift-network-diagnostics/network-check-target-7cjjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:15.870883 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.870655    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret podName:3e73b381-beec-43fe-ba3d-460efc605f8b nodeName:}" failed. No retries permitted until 2026-11-03 23:28:16.870637081 +0000 UTC m=+4.103559759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret") pod "global-pull-secret-syncer-pcvlc" (UID: "3e73b381-beec-43fe-ba3d-460efc605f8b") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:15.870883 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:15.870675    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p podName:d2d422d3-fe51-4985-8577-4ddfcc24ba8c nodeName:}" failed. No retries permitted until 2026-11-03 23:28:16.870664822 +0000 UTC m=+4.103587514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8662p" (UniqueName: "kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p") pod "network-check-target-7cjjg" (UID: "d2d422d3-fe51-4985-8577-4ddfcc24ba8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:16.260726 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.260571    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:15 +0000 UTC" deadline="2028-08-10 05:17:07.857333628 +0000 UTC"
Nov 03 23:28:16.260726 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.260701    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15485h48m51.596638089s"
Nov 03 23:28:16.525969 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.525091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:16.525969 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.525243    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:16.525969 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.525702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:16.525969 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.525803    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:16.565685 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.565598    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerStarted","Data":"227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278"}
Nov 03 23:28:16.777808 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.777712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:16.777982 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.777887    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:16.778044 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.778024    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:18.777934008 +0000 UTC m=+6.010856678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:16.878378 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.878144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:16.878378 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:16.878196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:16.878378 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.878359    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:16.878378 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.878372    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:16.878378 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.878391    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:16.878774 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.878405    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8662p for pod openshift-network-diagnostics/network-check-target-7cjjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:16.878774 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.878434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret podName:3e73b381-beec-43fe-ba3d-460efc605f8b nodeName:}" failed. No retries permitted until 2026-11-03 23:28:18.878415082 +0000 UTC m=+6.111337753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret") pod "global-pull-secret-syncer-pcvlc" (UID: "3e73b381-beec-43fe-ba3d-460efc605f8b") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:16.878774 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:16.878451    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p podName:d2d422d3-fe51-4985-8577-4ddfcc24ba8c nodeName:}" failed. No retries permitted until 2026-11-03 23:28:18.87844435 +0000 UTC m=+6.111367022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8662p" (UniqueName: "kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p") pod "network-check-target-7cjjg" (UID: "d2d422d3-fe51-4985-8577-4ddfcc24ba8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:17.260968 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:17.260827    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:15 +0000 UTC" deadline="2028-04-30 16:25:38.381564986 +0000 UTC"
Nov 03 23:28:17.260968 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:17.260873    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13048h57m21.120698349s"
Nov 03 23:28:17.528681 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:17.527962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:17.528681 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:17.528274    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:17.581433 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:17.580639    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"14f459d1824c1ea630aa90a99306c647","Type":"ContainerStarted","Data":"08d36c0423e39b8aed45905bfc02f8eaddedadc910a336c904191d342816ef3d"}
Nov 03 23:28:18.526347 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.525591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:18.526347 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.525729    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:18.526347 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.526147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:18.526347 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.526261    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:18.599768 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.599724    2580 generic.go:356] "Generic (PLEG): container finished" podID="a9a45f1006b1620e684bf8ebed402834" containerID="b5ea30da3679c0290a89ecd50f10f220485a6d2038440c0198745f6b7b3b080f" exitCode=0
Nov 03 23:28:18.599962 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.599900    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"a9a45f1006b1620e684bf8ebed402834","Type":"ContainerDied","Data":"b5ea30da3679c0290a89ecd50f10f220485a6d2038440c0198745f6b7b3b080f"}
Nov 03 23:28:18.620249 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.619327    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" podStartSLOduration=4.619307286 podStartE2EDuration="4.619307286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:17.599327194 +0000 UTC m=+4.832249903" watchObservedRunningTime="2026-11-03 23:28:18.619307286 +0000 UTC m=+5.852229981"
Nov 03 23:28:18.798515 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.796884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:18.798515 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.797086    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:18.798515 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.797157    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:22.79713743 +0000 UTC m=+10.030060116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.898011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:18.898111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.898229    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.898292    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret podName:3e73b381-beec-43fe-ba3d-460efc605f8b nodeName:}" failed. No retries permitted until 2026-11-03 23:28:22.898273403 +0000 UTC m=+10.131196101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret") pod "global-pull-secret-syncer-pcvlc" (UID: "3e73b381-beec-43fe-ba3d-460efc605f8b") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.898382    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.898399    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.898411    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8662p for pod openshift-network-diagnostics/network-check-target-7cjjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:18.898929 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:18.898452    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p podName:d2d422d3-fe51-4985-8577-4ddfcc24ba8c nodeName:}" failed. No retries permitted until 2026-11-03 23:28:22.898438987 +0000 UTC m=+10.131361664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8662p" (UniqueName: "kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p") pod "network-check-target-7cjjg" (UID: "d2d422d3-fe51-4985-8577-4ddfcc24ba8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:19.528269 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:19.525517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:19.528269 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:19.525688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:19.613376 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:19.612596    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"a9a45f1006b1620e684bf8ebed402834","Type":"ContainerStarted","Data":"0350f1acdb7dfec8473f5e05475f587c299316f40567460b5bcb10ebe5de03f1"}
Nov 03 23:28:19.627509 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:19.624033    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="e8009b352aa2a547ecb311d91f279f083bdbccb06b496a4256bcdf7bca98bfe1" exitCode=0
Nov 03 23:28:19.627509 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:19.624077    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerDied","Data":"e8009b352aa2a547ecb311d91f279f083bdbccb06b496a4256bcdf7bca98bfe1"}
Nov 03 23:28:19.633301 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:19.631497    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" podStartSLOduration=5.631480286 podStartE2EDuration="5.631480286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:19.630318184 +0000 UTC m=+6.863240879" watchObservedRunningTime="2026-11-03 23:28:19.631480286 +0000 UTC m=+6.864402977"
Nov 03 23:28:20.525837 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:20.525794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:20.526119 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:20.525950    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:20.526432 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:20.526383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:20.526525 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:20.526500    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:20.637538 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:20.637503    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b9jnr" event={"ID":"47845d6a-a330-49f4-be05-d3a8aeb677dc","Type":"ContainerStarted","Data":"897422a250eabbc453fdba6ecede1a07716589c6cc979edc1a2c65fdef89daf9"}
Nov 03 23:28:21.526682 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:21.526187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:21.526682 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:21.526329    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:21.646664 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:21.646308    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cdxlq" event={"ID":"cd9bbc11-d972-4662-bb0d-b1d8744673c9","Type":"ContainerStarted","Data":"82c59874976a582499d6347056c4c4dcfee66fa69010b2952631cf8efee27245"}
Nov 03 23:28:21.663626 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:21.662936    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b9jnr" podStartSLOduration=3.298861384 podStartE2EDuration="7.66291689s" totalImagesPullingTime="4.364055506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:20.653664296 +0000 UTC m=+7.886586984" watchObservedRunningTime="2026-11-03 23:28:21.66291689 +0000 UTC m=+8.895839582"
Nov 03 23:28:21.929436 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:21.928874    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:21.929774 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:21.929730    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:21.946371 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:21.945436    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cdxlq" podStartSLOduration=3.445539849 podStartE2EDuration="8.945419113s" totalImagesPullingTime="5.499879264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:21.663593251 +0000 UTC m=+8.896515944" watchObservedRunningTime="2026-11-03 23:28:21.945419113 +0000 UTC m=+9.178341808"
Nov 03 23:28:22.525576 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:22.524966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:22.525576 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.525088    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:22.525576 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:22.525416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:22.525576 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.525513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:22.650633 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:22.650102    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4" event={"ID":"63593602-b03b-47c5-9d74-a95ff2abfddb","Type":"ContainerStarted","Data":"14af282bfdf0adf4d7610469d7ebbf1916eef65410ad13fedb9405dd89efe4a8"}
Nov 03 23:28:22.836508 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:22.835903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:22.836508 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.836062    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:22.836508 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.836124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:30.836106759 +0000 UTC m=+18.069029443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:22.936206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:22.936257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.936434    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.936465    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.936478    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8662p for pod openshift-network-diagnostics/network-check-target-7cjjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.936533    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p podName:d2d422d3-fe51-4985-8577-4ddfcc24ba8c nodeName:}" failed. No retries permitted until 2026-11-03 23:28:30.936516895 +0000 UTC m=+18.169439580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8662p" (UniqueName: "kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p") pod "network-check-target-7cjjg" (UID: "d2d422d3-fe51-4985-8577-4ddfcc24ba8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.936945    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:22.937063 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:22.937015    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret podName:3e73b381-beec-43fe-ba3d-460efc605f8b nodeName:}" failed. No retries permitted until 2026-11-03 23:28:30.93698882 +0000 UTC m=+18.169911489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret") pod "global-pull-secret-syncer-pcvlc" (UID: "3e73b381-beec-43fe-ba3d-460efc605f8b") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:23.181559 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.181532    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 23:28:23.212085 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.211978    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T23:28:23.181556504Z","UUID":"6658e898-f926-4617-9a6c-1600f72fd692","Handler":null,"Name":"","Endpoint":""}
Nov 03 23:28:23.218003 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.217977    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 23:28:23.218123 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.218019    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 23:28:23.529112 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.528562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:23.529112 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:23.528685    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:23.655549 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.655509    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:23.656018 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:23.655982    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4" event={"ID":"63593602-b03b-47c5-9d74-a95ff2abfddb","Type":"ContainerStarted","Data":"d1570c32e420a58e0da21f7ff549f5f64553f01629036a56ae6e44f36bd7f531"}
Nov 03 23:28:24.525480 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.525238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:24.526095 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:24.526053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:24.529804 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.525448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:24.529988 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:24.529963    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:24.661338 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.661253    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4" event={"ID":"63593602-b03b-47c5-9d74-a95ff2abfddb","Type":"ContainerStarted","Data":"72b9e8a8ba50b892f3b33e82deb106654c5e5849529b8ccee5f87c53517f8595"}
Nov 03 23:28:24.663436 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.663406    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="440884e102042d8b584a393b636d7bcf42b53e53d46237fd7df2608bb9468bbf" exitCode=0
Nov 03 23:28:24.663566 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.663495    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerDied","Data":"440884e102042d8b584a393b636d7bcf42b53e53d46237fd7df2608bb9468bbf"}
Nov 03 23:28:24.665391 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.665361    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt2sr" event={"ID":"a1e7a5aa-87e3-4afd-ae59-801a559622f6","Type":"ContainerStarted","Data":"a8294ff6f17deb9445cd912a2167dbabcc61522d1dfbc4476b9c0afadd40fc04"}
Nov 03 23:28:24.667356 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.667332    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c99qk" event={"ID":"c4738a53-824f-4a8b-ab86-b160b9f14c0e","Type":"ContainerStarted","Data":"4d58877eee20d71cec3d30307415a8d1faffd0f8e0c28a3e78cbb0519558ef1b"}
Nov 03 23:28:24.685050 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.685008    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4" podStartSLOduration=2.210235166 podStartE2EDuration="10.684996285s" totalImagesPullingTime="8.474761119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:24.684927452 +0000 UTC m=+11.917850143" watchObservedRunningTime="2026-11-03 23:28:24.684996285 +0000 UTC m=+11.917918976"
Nov 03 23:28:24.723377 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.723328    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c99qk" podStartSLOduration=3.329831319 podStartE2EDuration="11.723315276s" totalImagesPullingTime="8.393483957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:24.701984984 +0000 UTC m=+11.934907675" watchObservedRunningTime="2026-11-03 23:28:24.723315276 +0000 UTC m=+11.956237966"
Nov 03 23:28:24.723646 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:24.723613    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jt2sr" podStartSLOduration=2.55170791 podStartE2EDuration="10.723604504s" totalImagesPullingTime="8.171896594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:24.721633602 +0000 UTC m=+11.954556294" watchObservedRunningTime="2026-11-03 23:28:24.723604504 +0000 UTC m=+11.956527196"
Nov 03 23:28:25.525691 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:25.525662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:25.525844 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:25.525821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:25.672412 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:25.672382    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="1d526e5703b562cef09912248098bb924efbb325ecc77d6eb8951f517a227db6" exitCode=0
Nov 03 23:28:25.673516 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:25.673490    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerDied","Data":"1d526e5703b562cef09912248098bb924efbb325ecc77d6eb8951f517a227db6"}
Nov 03 23:28:26.524968 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:26.524904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:26.525112 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:26.525011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:26.525112 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:26.525056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:26.525187 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:26.525133    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:26.676261 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:26.676226    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxj77" event={"ID":"3a0dae53-7734-414d-94ca-7bfd0bbc5d62","Type":"ContainerStarted","Data":"a68942537111a69fcf56c8d5bff42376c768d5bbe8a684b2325b2a59c73f8e70"}
Nov 03 23:28:26.678528 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:26.678161    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6czcz" event={"ID":"fcd47e4a-ca1b-4557-82d0-200bdb384976","Type":"ContainerStarted","Data":"a40199bd8af71d4fbc97a9252ad21908cd6201c38122f7c703cb3598989c5519"}
Nov 03 23:28:26.695716 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:26.695676    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dxj77" podStartSLOduration=2.000414093 podStartE2EDuration="12.695663541s" totalImagesPullingTime="10.695249448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:26.695243364 +0000 UTC m=+13.928166055" watchObservedRunningTime="2026-11-03 23:28:26.695663541 +0000 UTC m=+13.928586232"
Nov 03 23:28:26.710985 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:26.710949    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6czcz" podStartSLOduration=4.099688062 podStartE2EDuration="12.710936739s" totalImagesPullingTime="8.611248677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:26.71063408 +0000 UTC m=+13.943556820" watchObservedRunningTime="2026-11-03 23:28:26.710936739 +0000 UTC m=+13.943859430"
Nov 03 23:28:27.526234 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:27.525754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:27.526234 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:27.525893    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:27.683207 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:27.683169    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="b12e05776bd12a9bef0293ea01409cbbf7a83f03d1d60768b3ca1515e45e4ae4" exitCode=0
Nov 03 23:28:27.683628 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:27.683255    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerDied","Data":"b12e05776bd12a9bef0293ea01409cbbf7a83f03d1d60768b3ca1515e45e4ae4"}
Nov 03 23:28:28.526327 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:28.525447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:28.526327 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:28.525825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:28.526327 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:28.525448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:28.526327 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:28.526226    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:29.525178 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:29.525154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:29.525589 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:29.525258    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:29.690481 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:29.690450    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="e16ea40b2ec00f27e88099f72e0f8c55790dfe7e47c62612a2fc19767767d75f" exitCode=0
Nov 03 23:28:29.690600 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:29.690505    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerDied","Data":"e16ea40b2ec00f27e88099f72e0f8c55790dfe7e47c62612a2fc19767767d75f"}
Nov 03 23:28:30.525635 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.525599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:30.526061 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.525640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:30.526061 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.525758    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:30.526061 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.525864    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:30.695138 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.695107    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="9a08842926263ee132242508a6c9c30af474d539f787e9ad58f665949ca6a6c6" exitCode=0
Nov 03 23:28:30.695327 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.695160    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerDied","Data":"9a08842926263ee132242508a6c9c30af474d539f787e9ad58f665949ca6a6c6"}
Nov 03 23:28:30.897926 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.897677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:30.898026 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.897828    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:30.898092 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.898074    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:46.898050865 +0000 UTC m=+34.130973551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:30.998584 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.998560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:30.998689 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:30.998594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:30.998770 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.998717    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:30.998770 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.998724    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:30.998844 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.998811    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret podName:3e73b381-beec-43fe-ba3d-460efc605f8b nodeName:}" failed. No retries permitted until 2026-11-03 23:28:46.998790714 +0000 UTC m=+34.231713397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret") pod "global-pull-secret-syncer-pcvlc" (UID: "3e73b381-beec-43fe-ba3d-460efc605f8b") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:30.998844 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.998730    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:30.998844 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.998835    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8662p for pod openshift-network-diagnostics/network-check-target-7cjjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:30.998943 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:30.998887    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p podName:d2d422d3-fe51-4985-8577-4ddfcc24ba8c nodeName:}" failed. No retries permitted until 2026-11-03 23:28:46.998871222 +0000 UTC m=+34.231793906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8662p" (UniqueName: "kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p") pod "network-check-target-7cjjg" (UID: "d2d422d3-fe51-4985-8577-4ddfcc24ba8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:31.525575 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.525542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:31.525802 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:31.525683    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:31.699626 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.699587    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"89ae361930570cbc5b27365b5c38c203263d79d2835687024890727373145b25"}
Nov 03 23:28:31.699782 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.699655    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"3062f9701ecbdc6b5444753d60c210c8406ad402b66ad28ddabec78efe7ecb65"}
Nov 03 23:28:31.699782 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.699697    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"e0c39570dd3ecb92ab46555f2e1e1ef560bf6250aba2daf9324b153cd3aec4a0"}
Nov 03 23:28:31.699782 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.699728    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"dc7642d50c7677208b3546a753c8c6a6120efbd2e7380ecb80894f097b8f36d0"}
Nov 03 23:28:31.699931 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.699783    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"f80d3aae29826861d69048ea8f8808c17e3855610742246b801b38233c330e17"}
Nov 03 23:28:31.699931 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.699820    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"ba15dd8d64d887061796522f506d1c847440c461cd704fccace99bcb4cca82fb"}
Nov 03 23:28:31.702761 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.702727    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" event={"ID":"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc","Type":"ContainerStarted","Data":"6114c3a0c9c56ec6796930164ff4dca174e65693a9b70b16af884140ecae801f"}
Nov 03 23:28:31.728394 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:31.728350    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cqzg8" podStartSLOduration=6.8598990109999995 podStartE2EDuration="17.728339154s" totalImagesPullingTime="10.868440143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:31.727888487 +0000 UTC m=+18.960811178" watchObservedRunningTime="2026-11-03 23:28:31.728339154 +0000 UTC m=+18.961261845"
Nov 03 23:28:32.525560 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:32.525528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:32.525560 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:32.525564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:32.525726 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:32.525640    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:32.525804 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:32.525781    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:33.526116 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:33.526082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:33.526698 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:33.526168    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:33.709654 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:33.709620    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"064fa5cac1d516fb3a222ad9aa3477a749efc32991afa2844ad6c77cdf6503e9"}
Nov 03 23:28:34.525133 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:34.525101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:34.525133 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:34.525136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:34.525308 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:34.525219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:34.525364 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:34.525335    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:35.525798 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:35.525612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:35.526413 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:35.525906    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:35.717087 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:35.717056    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" event={"ID":"5bc8021d-2812-40f5-99ba-85e10f04ef4f","Type":"ContainerStarted","Data":"5c3f19ed531574af554f2775410ee14acb8b8ed3260dac422cd71384224d3d1e"}
Nov 03 23:28:35.717322 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:35.717303    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:35.732632 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:35.732610    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:35.748315 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:35.748274    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n" podStartSLOduration=7.517979594 podStartE2EDuration="22.748260042s" totalImagesPullingTime="15.230280448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:35.747771689 +0000 UTC m=+22.980694381" watchObservedRunningTime="2026-11-03 23:28:35.748260042 +0000 UTC m=+22.981182734"
Nov 03 23:28:36.524828 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.524794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:36.524828 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.524834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:36.525108 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:36.524914    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:36.525108 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:36.525037    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:36.719860 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.719836    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:36.720286 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.720268    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:36.734904 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.734880    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:36.944214 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.944189    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7cjjg"]
Nov 03 23:28:36.944364 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.944284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:36.944414 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:36.944361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:36.945107 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.945081    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcvlc"]
Nov 03 23:28:36.945225 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.945190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:36.945300 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:36.945278    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:36.947083 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.947064    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6n2fd"]
Nov 03 23:28:36.947167 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:36.947136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:36.947227 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:36.947204    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:37.721694 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:37.721666    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:38.524815 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:38.524788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:38.524984 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:38.524835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:38.524984 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:38.524930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:38.524984 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:38.524935    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:38.525146 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:38.525056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:38.525197 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:38.525143    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:38.723604 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:38.723581    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:40.525062 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:40.525030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:40.525552 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:40.525071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:40.525552 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:40.525125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:40.525552 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:40.525136    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:40.525552 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:40.525235    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:40.525552 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:40.525311    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:42.486722 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:42.486692    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:42.487177 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:42.486945    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:42.496724 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:42.496704    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:42.525641 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:42.525620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:42.525769 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:42.525647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:42.525769 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:42.525690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:42.525769 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:42.525699    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcvlc" podUID="3e73b381-beec-43fe-ba3d-460efc605f8b"
Nov 03 23:28:42.525935 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:42.525802    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6n2fd" podUID="1244c649-9fe7-40c5-9b55-443a2529c710"
Nov 03 23:28:42.525935 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:42.525892    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7cjjg" podUID="d2d422d3-fe51-4985-8577-4ddfcc24ba8c"
Nov 03 23:28:43.504255 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.504083    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:43.504663 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.504361    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:43.504917 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.504895    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:43.536168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.536148    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeReady"
Nov 03 23:28:43.536293 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.536247    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 23:28:43.574044 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.574012    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6mxbn"]
Nov 03 23:28:43.576526 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.576508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:43.578711 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.578691    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c6qhd"]
Nov 03 23:28:43.581599 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.581577    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qjdfr"]
Nov 03 23:28:43.582308 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.582289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:43.586171 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.586081    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"]
Nov 03 23:28:43.586398 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.586367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:43.593687 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.593664    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bb58b79c9-bksx8"]
Nov 03 23:28:43.593939 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.593912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.594177 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.594102    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.594177 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.594151    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.596989 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.596969    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7869h"]
Nov 03 23:28:43.597210 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.597176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.600019 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.600003    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"]
Nov 03 23:28:43.600155 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.600135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.604888 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.604872    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"]
Nov 03 23:28:43.605016 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.604979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.607900 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.607882    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6768769b75-2ms22"]
Nov 03 23:28:43.608004 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.607990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.610719 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.610703    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"]
Nov 03 23:28:43.610837 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.610802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.614009 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.613993    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"]
Nov 03 23:28:43.614117 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.614104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.616804 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.616786    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"]
Nov 03 23:28:43.616900 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.616888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:43.622433 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.622410    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 23:28:43.622801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.622766    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 23:28:43.622912 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.622851    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 23:28:43.623191 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623168    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 23:28:43.623272 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623250    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9j4q5\""
Nov 03 23:28:43.623368 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623349    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nk9gg\""
Nov 03 23:28:43.623472 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623392    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 23:28:43.623472 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623433    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 23:28:43.623582 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623544    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-78rrm\""
Nov 03 23:28:43.623810 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.623791    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zb5ww\""
Nov 03 23:28:43.631527 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.627004    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"]
Nov 03 23:28:43.631527 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.627083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:43.632799 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.632781    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b96cb8d6-ctr95"]
Nov 03 23:28:43.632909 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.632839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.637843 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.637826    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"]
Nov 03 23:28:43.637977 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.637960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.646617 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.646595    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6mxbn"]
Nov 03 23:28:43.646733 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.646643    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c6qhd"]
Nov 03 23:28:43.646733 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.646667    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4x94t"]
Nov 03 23:28:43.646961 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.646943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:43.649634 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.649613    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6mthf"]
Nov 03 23:28:43.650531 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.649843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.653768 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653718    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"]
Nov 03 23:28:43.653885 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653780    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"]
Nov 03 23:28:43.653885 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653804    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"]
Nov 03 23:28:43.653885 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653823    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"]
Nov 03 23:28:43.653885 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653843    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"]
Nov 03 23:28:43.653885 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653876    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb58b79c9-bksx8"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653897    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653917    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4x94t"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653948    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qjdfr"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653966    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.653985    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6768769b75-2ms22"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.654004    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6mthf"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.654077    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b96cb8d6-ctr95"]
Nov 03 23:28:43.654213 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.654157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.654657 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.654637    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"]
Nov 03 23:28:43.657837 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.657811    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7869h"]
Nov 03 23:28:43.687826 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsv66\" (UniqueName: \"kubernetes.io/projected/5032904c-5a18-4f28-be3b-a02375b14113-kube-api-access-wsv66\") pod \"kas-connection-checker-685895c7c9-c6qhd\" (UID: \"5032904c-5a18-4f28-be3b-a02375b14113\") " pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:43.687954 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-tmp\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.687954 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-service-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.687954 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.687954 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmv7\" (UniqueName: \"kubernetes.io/projected/bbbb34c1-9e43-4640-82ae-660e1e365930-kube-api-access-6kmv7\") pod \"kas-connection-checker-685895c7c9-6mxbn\" (UID: \"bbbb34c1-9e43-4640-82ae-660e1e365930\") " pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:43.688179 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.688179 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.687992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.688179 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-default-certificate\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.688179 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59s2\" (UniqueName: \"kubernetes.io/projected/13b0dc51-f5cc-4afd-821f-c3097be21e80-kube-api-access-c59s2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.688179 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.688179 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13b0dc51-f5cc-4afd-821f-c3097be21e80-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-snapshots\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6ff\" (UniqueName: \"kubernetes.io/projected/69d478d4-7d64-4a68-823e-29c6b40b4a94-kube-api-access-kp6ff\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13b0dc51-f5cc-4afd-821f-c3097be21e80-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhwq\" (UniqueName: \"kubernetes.io/projected/40135659-f74d-4307-8bb3-574411040f5a-kube-api-access-khhwq\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.688498 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d478d4-7d64-4a68-823e-29c6b40b4a94-serving-cert\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.688937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twwh\" (UniqueName: \"kubernetes.io/projected/698112a3-0157-43a7-9ea6-30e554d8be51-kube-api-access-9twwh\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.688937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.688937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmx5\" (UniqueName: \"kubernetes.io/projected/59264120-38b0-4c6d-b3be-7d0340db0e34-kube-api-access-bcmx5\") pod \"volume-data-source-validator-58c9f5f46d-svxq9\" (UID: \"59264120-38b0-4c6d-b3be-7d0340db0e34\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:43.688937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vm4\" (UniqueName: \"kubernetes.io/projected/a1f0dd5d-1903-4b2a-830c-05e93d384fda-kube-api-access-k6vm4\") pod \"kas-connection-checker-685895c7c9-qjdfr\" (UID: \"a1f0dd5d-1903-4b2a-830c-05e93d384fda\") " pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:43.688937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.688937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.689238 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.688973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-serving-cert\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.689238 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.689238 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-stats-auth\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.689238 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-config\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.689384 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4741d3bf-14a8-4118-a61e-32dd236755dd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.689384 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdbc\" (UniqueName: \"kubernetes.io/projected/4741d3bf-14a8-4118-a61e-32dd236755dd-kube-api-access-fkdbc\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.689384 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.689384 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g86bg\" (UniqueName: \"kubernetes.io/projected/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-kube-api-access-g86bg\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.689493 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.689399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.695923 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.695902    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 23:28:43.696019 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.695979    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 23:28:43.696086 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696061    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.696168 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696151    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.696225 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696166    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.696225 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696167    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.696669 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696651    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.696669 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696656    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696677    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696690    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696707    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696658    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696725    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696734    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696755    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696793    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696809    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.696832 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696832    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696857    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696868    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696880    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696915    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.696965    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.697146    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 23:28:43.697192 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.697192    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.697426 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.697212    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 23:28:43.697426 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.697223    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 23:28:43.697426 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.697385    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 23:28:43.699200 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.699184    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 23:28:43.700449 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.700429    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 23:28:43.700727 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.700705    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 23:28:43.702484 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.702456    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 23:28:43.724218 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724199    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 23:28:43.724218 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724200    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 23:28:43.724382 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724312    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 23:28:43.724382 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724313    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 23:28:43.724476 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724424    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 23:28:43.724476 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724448    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 23:28:43.724568 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724471    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 23:28:43.724568 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724482    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k24zf\""
Nov 03 23:28:43.724568 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724487    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ws2sq\""
Nov 03 23:28:43.724568 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724480    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 23:28:43.724568 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724549    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kpv5j\""
Nov 03 23:28:43.724801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724576    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t7fl4\""
Nov 03 23:28:43.724801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724652    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-swp8v\""
Nov 03 23:28:43.724801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724664    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xj6sl\""
Nov 03 23:28:43.724801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724670    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z6886\""
Nov 03 23:28:43.724801 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724683    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k4d88\""
Nov 03 23:28:43.724937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.724826    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 23:28:43.725108 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.725091    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q2zjl\""
Nov 03 23:28:43.789850 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.789791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4741d3bf-14a8-4118-a61e-32dd236755dd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.789850 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.789817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdbc\" (UniqueName: \"kubernetes.io/projected/4741d3bf-14a8-4118-a61e-32dd236755dd-kube-api-access-fkdbc\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.790044 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.789856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.790044 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.789873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g86bg\" (UniqueName: \"kubernetes.io/projected/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-kube-api-access-g86bg\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.790044 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.789892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.790044 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.789968    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:43.790044 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.790018    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls podName:4741d3bf-14a8-4118-a61e-32dd236755dd nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.290005213 +0000 UTC m=+31.522927882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ngdxl" (UID: "4741d3bf-14a8-4118-a61e-32dd236755dd") : secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:43.790221 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.790221 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.790221 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4zq\" (UniqueName: \"kubernetes.io/projected/17673e8b-0dec-48d4-8843-65147e22372a-kube-api-access-vw4zq\") pod \"network-check-source-6f78d59f99-xr79v\" (UID: \"17673e8b-0dec-48d4-8843-65147e22372a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:43.790221 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:43.790342 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsv66\" (UniqueName: \"kubernetes.io/projected/5032904c-5a18-4f28-be3b-a02375b14113-kube-api-access-wsv66\") pod \"kas-connection-checker-685895c7c9-c6qhd\" (UID: \"5032904c-5a18-4f28-be3b-a02375b14113\") " pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:43.790342 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b107666-eaaa-4d08-839b-d3e91dd5868b-cert\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.790342 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.790342 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-tmp\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.790529 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-service-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.790529 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.790529 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmv7\" (UniqueName: \"kubernetes.io/projected/bbbb34c1-9e43-4640-82ae-660e1e365930-kube-api-access-6kmv7\") pod \"kas-connection-checker-685895c7c9-6mxbn\" (UID: \"bbbb34c1-9e43-4640-82ae-660e1e365930\") " pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:43.790661 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.790661 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.790661 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.790661 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.790868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790668    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-tmp\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.790868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.790868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-serving-cert\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.790868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-default-certificate\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.790868 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59s2\" (UniqueName: \"kubernetes.io/projected/13b0dc51-f5cc-4afd-821f-c3097be21e80-kube-api-access-c59s2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790826    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4741d3bf-14a8-4118-a61e-32dd236755dd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790921    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-service-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.790961    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.790974    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.790982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.791028    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.291011951 +0000 UTC m=+31.523934634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13b0dc51-f5cc-4afd-821f-c3097be21e80-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.791089 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-snapshots\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6ff\" (UniqueName: \"kubernetes.io/projected/69d478d4-7d64-4a68-823e-29c6b40b4a94-kube-api-access-kp6ff\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8lk\" (UniqueName: \"kubernetes.io/projected/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-kube-api-access-gc8lk\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791235    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13b0dc51-f5cc-4afd-821f-c3097be21e80-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhwq\" (UniqueName: \"kubernetes.io/projected/40135659-f74d-4307-8bb3-574411040f5a-kube-api-access-khhwq\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d478d4-7d64-4a68-823e-29c6b40b4a94-serving-cert\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vfw\" (UniqueName: \"kubernetes.io/projected/3b107666-eaaa-4d08-839b-d3e91dd5868b-kube-api-access-g8vfw\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-config\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm94g\" (UniqueName: \"kubernetes.io/projected/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-kube-api-access-mm94g\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.791615 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791486    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twwh\" (UniqueName: \"kubernetes.io/projected/698112a3-0157-43a7-9ea6-30e554d8be51-kube-api-access-9twwh\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791497    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791623    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13b0dc51-f5cc-4afd-821f-c3097be21e80-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791704    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-snapshots\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.791780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.791846    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.791902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.29188863 +0000 UTC m=+31.524811299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : secret "router-metrics-certs-default" not found
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.791918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.291910111 +0000 UTC m=+31.524832785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : configmap references non-existent config key: service-ca.crt
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-tmp-dir\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmx5\" (UniqueName: \"kubernetes.io/projected/59264120-38b0-4c6d-b3be-7d0340db0e34-kube-api-access-bcmx5\") pod \"volume-data-source-validator-58c9f5f46d-svxq9\" (UID: \"59264120-38b0-4c6d-b3be-7d0340db0e34\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vm4\" (UniqueName: \"kubernetes.io/projected/a1f0dd5d-1903-4b2a-830c-05e93d384fda-kube-api-access-k6vm4\") pod \"kas-connection-checker-685895c7c9-qjdfr\" (UID: \"a1f0dd5d-1903-4b2a-830c-05e93d384fda\") " pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.792294 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-serving-cert\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.792976 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.792339    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:43.792976 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.792383    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.292368516 +0000 UTC m=+31.525291186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:43.793069 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.792993    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.793155 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.793134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.793208 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.793189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-config-volume\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.793831 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.793807    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.794984 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.794584    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.794984 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.794671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-stats-auth\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.794984 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.794832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-config\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.795261 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795013    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d478d4-7d64-4a68-823e-29c6b40b4a94-serving-cert\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.795261 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795187    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.795261 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795217    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-serving-cert\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.795261 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795225    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13b0dc51-f5cc-4afd-821f-c3097be21e80-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.795261 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795249    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.795548 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795529    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-config\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.795713 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.795695    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-default-certificate\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.797162 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.797147    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-stats-auth\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.805473 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.805454    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6ff\" (UniqueName: \"kubernetes.io/projected/69d478d4-7d64-4a68-823e-29c6b40b4a94-kube-api-access-kp6ff\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.805756 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.805721    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdbc\" (UniqueName: \"kubernetes.io/projected/4741d3bf-14a8-4118-a61e-32dd236755dd-kube-api-access-fkdbc\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:43.805847 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.805825    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsv66\" (UniqueName: \"kubernetes.io/projected/5032904c-5a18-4f28-be3b-a02375b14113-kube-api-access-wsv66\") pod \"kas-connection-checker-685895c7c9-c6qhd\" (UID: \"5032904c-5a18-4f28-be3b-a02375b14113\") " pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:43.807982 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.807962    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.808072 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.807965    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vm4\" (UniqueName: \"kubernetes.io/projected/a1f0dd5d-1903-4b2a-830c-05e93d384fda-kube-api-access-k6vm4\") pod \"kas-connection-checker-685895c7c9-qjdfr\" (UID: \"a1f0dd5d-1903-4b2a-830c-05e93d384fda\") " pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:43.808072 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.808011    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twwh\" (UniqueName: \"kubernetes.io/projected/698112a3-0157-43a7-9ea6-30e554d8be51-kube-api-access-9twwh\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:43.811113 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.811074    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmx5\" (UniqueName: \"kubernetes.io/projected/59264120-38b0-4c6d-b3be-7d0340db0e34-kube-api-access-bcmx5\") pod \"volume-data-source-validator-58c9f5f46d-svxq9\" (UID: \"59264120-38b0-4c6d-b3be-7d0340db0e34\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:43.811576 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.811556    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmv7\" (UniqueName: \"kubernetes.io/projected/bbbb34c1-9e43-4640-82ae-660e1e365930-kube-api-access-6kmv7\") pod \"kas-connection-checker-685895c7c9-6mxbn\" (UID: \"bbbb34c1-9e43-4640-82ae-660e1e365930\") " pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:43.812108 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.812087    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhwq\" (UniqueName: \"kubernetes.io/projected/40135659-f74d-4307-8bb3-574411040f5a-kube-api-access-khhwq\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:43.813156 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.813137    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g86bg\" (UniqueName: \"kubernetes.io/projected/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-kube-api-access-g86bg\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.813414 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.813396    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:43.813906 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.813883    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59s2\" (UniqueName: \"kubernetes.io/projected/13b0dc51-f5cc-4afd-821f-c3097be21e80-kube-api-access-c59s2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.895856 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.895831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.896006 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.895862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.896006 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.895880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.896006 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.895897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-serving-cert\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.896006 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.895992    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:43.896006 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.896011    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:43.896256 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.896079    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.396059443 +0000 UTC m=+31.628982134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:43.896348 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.896453 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.896645 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.896779 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8lk\" (UniqueName: \"kubernetes.io/projected/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-kube-api-access-gc8lk\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.896779 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.896761    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:43.896890 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.896832    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.396816187 +0000 UTC m=+31.629738872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:43.896890 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vfw\" (UniqueName: \"kubernetes.io/projected/3b107666-eaaa-4d08-839b-d3e91dd5868b-kube-api-access-g8vfw\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.896890 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-config\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.897036 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896889    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.897036 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm94g\" (UniqueName: \"kubernetes.io/projected/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-kube-api-access-mm94g\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.897036 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.896955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:43.897036 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-tmp-dir\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.897221 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.897115    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 23:28:43.897221 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:43.897165    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert podName:dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d nodeName:}" failed. No retries permitted until 2026-11-03 23:28:44.39715059 +0000 UTC m=+31.630073273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cs8jk" (UID: "dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d") : secret "networking-console-plugin-cert" not found
Nov 03 23:28:43.897221 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-config-volume\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.897363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.897363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.897363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4zq\" (UniqueName: \"kubernetes.io/projected/17673e8b-0dec-48d4-8843-65147e22372a-kube-api-access-vw4zq\") pod \"network-check-source-6f78d59f99-xr79v\" (UID: \"17673e8b-0dec-48d4-8843-65147e22372a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:43.897363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897328    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-tmp-dir\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.897363 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:43.897622 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897466    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-config\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.897715 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b107666-eaaa-4d08-839b-d3e91dd5868b-cert\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.897715 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.897983 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.897959    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-config-volume\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.898059 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.898042    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.898601 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.898575    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-serving-cert\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.898697 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.898603    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:43.898780 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.898720    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.898900 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.898884    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.900255 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.900235    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.900610 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.900595    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b107666-eaaa-4d08-839b-d3e91dd5868b-cert\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.901903 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.901887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:43.904800 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.904783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:43.910671 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.910645    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8lk\" (UniqueName: \"kubernetes.io/projected/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-kube-api-access-gc8lk\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:43.910767 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.910717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:43.911242 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.911218    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.912798 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.912768    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm94g\" (UniqueName: \"kubernetes.io/projected/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-kube-api-access-mm94g\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:43.912888 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.912867    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:43.913311 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.913291    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4zq\" (UniqueName: \"kubernetes.io/projected/17673e8b-0dec-48d4-8843-65147e22372a-kube-api-access-vw4zq\") pod \"network-check-source-6f78d59f99-xr79v\" (UID: \"17673e8b-0dec-48d4-8843-65147e22372a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:43.913387 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.913372    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vfw\" (UniqueName: \"kubernetes.io/projected/3b107666-eaaa-4d08-839b-d3e91dd5868b-kube-api-access-g8vfw\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:43.937682 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.937629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:43.944884 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.944852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:43.977291 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.976685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:43.979207 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.979171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:43.995679 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:43.994689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:44.017084 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.012852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:44.025875 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.025855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:44.221002 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.220957    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6mxbn"]
Nov 03 23:28:44.225498 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.225465    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbb34c1_9e43_4640_82ae_660e1e365930.slice/crio-701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925 WatchSource:0}: Error finding container 701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925: Status 404 returned error can't find the container with id 701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925
Nov 03 23:28:44.227022 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.227001    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qjdfr"]
Nov 03 23:28:44.229210 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.227915    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f0dd5d_1903_4b2a_830c_05e93d384fda.slice/crio-4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2 WatchSource:0}: Error finding container 4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2: Status 404 returned error can't find the container with id 4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2
Nov 03 23:28:44.229210 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.228091    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c6qhd"]
Nov 03 23:28:44.229210 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.228504    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5032904c_5a18_4f28_be3b_a02375b14113.slice/crio-942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850 WatchSource:0}: Error finding container 942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850: Status 404 returned error can't find the container with id 942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850
Nov 03 23:28:44.255632 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.255607    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7869h"]
Nov 03 23:28:44.260497 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.260472    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d478d4_7d64_4a68_823e_29c6b40b4a94.slice/crio-b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111 WatchSource:0}: Error finding container b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111: Status 404 returned error can't find the container with id b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111
Nov 03 23:28:44.282849 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.282824    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"]
Nov 03 23:28:44.288976 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.288948    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b0dc51_f5cc_4afd_821f_c3097be21e80.slice/crio-e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24 WatchSource:0}: Error finding container e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24: Status 404 returned error can't find the container with id e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24
Nov 03 23:28:44.303284 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.303257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:44.303387 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.303342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:44.303387 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.303372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:44.303487 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.303450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:44.303540 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.303511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:44.303691 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.303668    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:44.303791 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.303728    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls podName:4741d3bf-14a8-4118-a61e-32dd236755dd nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.303709117 +0000 UTC m=+32.536631785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ngdxl" (UID: "4741d3bf-14a8-4118-a61e-32dd236755dd") : secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:44.304162 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304125    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:44.304162 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304150    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:44.304409 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304195    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.30418017 +0000 UTC m=+32.537102840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:44.304409 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304257    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 23:28:44.304409 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304290    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.304279456 +0000 UTC m=+32.537202141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : secret "router-metrics-certs-default" not found
Nov 03 23:28:44.304409 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304357    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.30434418 +0000 UTC m=+32.537266849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : configmap references non-existent config key: service-ca.crt
Nov 03 23:28:44.304720 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304419    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:44.304720 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.304451    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.304440026 +0000 UTC m=+32.537362696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:44.324406 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.324344    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"]
Nov 03 23:28:44.329769 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.329731    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"]
Nov 03 23:28:44.335840 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.335806    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59264120_38b0_4c6d_b3be_7d0340db0e34.slice/crio-360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a WatchSource:0}: Error finding container 360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a: Status 404 returned error can't find the container with id 360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a
Nov 03 23:28:44.336072 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.336051    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e0a5aa_fe33_44ab_b2b9_811ddffea0fb.slice/crio-22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064 WatchSource:0}: Error finding container 22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064: Status 404 returned error can't find the container with id 22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064
Nov 03 23:28:44.361869 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.361844    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"]
Nov 03 23:28:44.365889 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.365860    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17673e8b_0dec_48d4_8843_65147e22372a.slice/crio-c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf WatchSource:0}: Error finding container c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf: Status 404 returned error can't find the container with id c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf
Nov 03 23:28:44.387534 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.387507    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"]
Nov 03 23:28:44.390143 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.390121    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb8c88a_1985_4ca8_a11f_7ab86cd87634.slice/crio-2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff WatchSource:0}: Error finding container 2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff: Status 404 returned error can't find the container with id 2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff
Nov 03 23:28:44.392974 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.392934    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6mthf"]
Nov 03 23:28:44.397176 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:44.397156    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b107666_eaaa_4d08_839b_d3e91dd5868b.slice/crio-535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1 WatchSource:0}: Error finding container 535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1: Status 404 returned error can't find the container with id 535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1
Nov 03 23:28:44.404726 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.404705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:44.404812 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.404783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:44.404862 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.404842    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:44.404862 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.404857    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:44.404930 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.404852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:44.404930 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.404908    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.404888863 +0000 UTC m=+32.637811554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:44.404930 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.404921    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:44.405027 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.404968    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.404953755 +0000 UTC m=+32.637876438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:44.405027 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.404967    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 23:28:44.405027 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:44.405006    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert podName:dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.404997718 +0000 UTC m=+32.637920405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cs8jk" (UID: "dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d") : secret "networking-console-plugin-cert" not found
Nov 03 23:28:44.525008 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.524972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:44.525008 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.524991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:44.525408 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.525068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:44.530022 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.529989    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 23:28:44.530138 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.530097    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\""
Nov 03 23:28:44.530257 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.530239    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\""
Nov 03 23:28:44.530325 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.530291    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 23:28:44.530325 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.530303    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\""
Nov 03 23:28:44.736857 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.736827    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6mthf" event={"ID":"3b107666-eaaa-4d08-839b-d3e91dd5868b","Type":"ContainerStarted","Data":"535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1"}
Nov 03 23:28:44.737880 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.737855    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9" event={"ID":"59264120-38b0-4c6d-b3be-7d0340db0e34","Type":"ContainerStarted","Data":"360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a"}
Nov 03 23:28:44.738925 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.738907    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc" event={"ID":"3cb8c88a-1985-4ca8-a11f-7ab86cd87634","Type":"ContainerStarted","Data":"2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff"}
Nov 03 23:28:44.739888 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.739867    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" event={"ID":"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb","Type":"ContainerStarted","Data":"22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064"}
Nov 03 23:28:44.740785 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.740767    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7869h" event={"ID":"69d478d4-7d64-4a68-823e-29c6b40b4a94","Type":"ContainerStarted","Data":"b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111"}
Nov 03 23:28:44.741963 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.741932    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qjdfr" event={"ID":"a1f0dd5d-1903-4b2a-830c-05e93d384fda","Type":"ContainerStarted","Data":"e7b6e6ec90bbce2004a252026ab1e5ed756901b2ddcda9bd1a700bdeb397e21f"}
Nov 03 23:28:44.742044 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.741973    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qjdfr" event={"ID":"a1f0dd5d-1903-4b2a-830c-05e93d384fda","Type":"ContainerStarted","Data":"4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2"}
Nov 03 23:28:44.743054 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.743036    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj" event={"ID":"13b0dc51-f5cc-4afd-821f-c3097be21e80","Type":"ContainerStarted","Data":"e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24"}
Nov 03 23:28:44.744397 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.744379    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6mxbn" event={"ID":"bbbb34c1-9e43-4640-82ae-660e1e365930","Type":"ContainerStarted","Data":"8ff4961c2f2a6353644d2d359bb1067a5d529524fb042c60af695dd673c0a86e"}
Nov 03 23:28:44.744478 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.744414    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6mxbn" event={"ID":"bbbb34c1-9e43-4640-82ae-660e1e365930","Type":"ContainerStarted","Data":"701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925"}
Nov 03 23:28:44.745562 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.745539    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v" event={"ID":"17673e8b-0dec-48d4-8843-65147e22372a","Type":"ContainerStarted","Data":"c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf"}
Nov 03 23:28:44.746901 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.746882    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c6qhd" event={"ID":"5032904c-5a18-4f28-be3b-a02375b14113","Type":"ContainerStarted","Data":"60ad2ce0e494e0b87715d52a1dae693fe24aa092f671c045aecd56ef8e0da989"}
Nov 03 23:28:44.746968 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.746912    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c6qhd" event={"ID":"5032904c-5a18-4f28-be3b-a02375b14113","Type":"ContainerStarted","Data":"942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850"}
Nov 03 23:28:44.759238 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.759193    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qjdfr" podStartSLOduration=456.759180314 podStartE2EDuration="7m36.759180314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:44.758677558 +0000 UTC m=+31.991600249" watchObservedRunningTime="2026-11-03 23:28:44.759180314 +0000 UTC m=+31.992103011"
Nov 03 23:28:44.774983 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.774950    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6mxbn" podStartSLOduration=456.774939635 podStartE2EDuration="7m36.774939635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:44.774412415 +0000 UTC m=+32.007335107" watchObservedRunningTime="2026-11-03 23:28:44.774939635 +0000 UTC m=+32.007862326"
Nov 03 23:28:44.797407 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:44.797340    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-c6qhd" podStartSLOduration=456.797330326 podStartE2EDuration="7m36.797330326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:44.797140505 +0000 UTC m=+32.030063198" watchObservedRunningTime="2026-11-03 23:28:44.797330326 +0000 UTC m=+32.030253017"
Nov 03 23:28:45.315191 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.315158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:45.315350 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.315217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:45.315350 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.315257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:45.315350 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.315310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:45.315350 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.315333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:45.315770 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.315466    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.315450327 +0000 UTC m=+34.548373010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : configmap references non-existent config key: service-ca.crt
Nov 03 23:28:45.315895 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.315876    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:45.315952 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.315942    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.315924944 +0000 UTC m=+34.548847628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:45.316020 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316010    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:45.316056 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316047    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls podName:4741d3bf-14a8-4118-a61e-32dd236755dd nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.316035716 +0000 UTC m=+34.548958386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ngdxl" (UID: "4741d3bf-14a8-4118-a61e-32dd236755dd") : secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:45.316164 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316111    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:45.316164 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316125    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:45.316252 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316169    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.316157727 +0000 UTC m=+34.549080410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:45.316252 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316229    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 23:28:45.316375 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.316259    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.316249172 +0000 UTC m=+34.549171856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : secret "router-metrics-certs-default" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.416264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.416405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:45.416501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.416636    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.416754    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.416720224 +0000 UTC m=+34.649642908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.417186    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.417251    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert podName:dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.417234495 +0000 UTC m=+34.650157166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cs8jk" (UID: "dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d") : secret "networking-console-plugin-cert" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.417317    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.417329    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:45.417448 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:45.417365    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:47.417352913 +0000 UTC m=+34.650275584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:46.932993 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:46.932436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:46.932993 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:46.932657    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 23:28:46.932993 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:46.932722    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:18.932701944 +0000 UTC m=+66.165624628 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : secret "metrics-daemon-secret" not found
Nov 03 23:28:47.034064 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.033331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:47.034064 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.033502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:47.051777 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.051657    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:47.053288 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.053193    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:47.267071 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.267027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:47.278273 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.278165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.336860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.336931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.336953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.337018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.337073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337111    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337134    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337192    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337202    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.337181714 +0000 UTC m=+38.570104404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337247    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls podName:4741d3bf-14a8-4118-a61e-32dd236755dd nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.337231758 +0000 UTC m=+38.570154441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ngdxl" (UID: "4741d3bf-14a8-4118-a61e-32dd236755dd") : secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337307    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337342    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.337330811 +0000 UTC m=+38.570253497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : secret "router-metrics-certs-default" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337402    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.337393368 +0000 UTC m=+38.570316068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : configmap references non-existent config key: service-ca.crt
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337455    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:47.337514 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.337486    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.337475471 +0000 UTC m=+38.570398178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.438412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.438484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.438555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.438690    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.438775    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert podName:dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.438754427 +0000 UTC m=+38.671677119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cs8jk" (UID: "dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d") : secret "networking-console-plugin-cert" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.438873    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.438898    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.438947    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.438929139 +0000 UTC m=+38.671851808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.439001    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:47.439255 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:47.439031    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.439023413 +0000 UTC m=+38.671946082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:47.575393 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.574489    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7cjjg"]
Nov 03 23:28:47.589250 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:47.589159    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d422d3_fe51_4985_8577_4ddfcc24ba8c.slice/crio-64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e WatchSource:0}: Error finding container 64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e: Status 404 returned error can't find the container with id 64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e
Nov 03 23:28:47.631673 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.631637    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcvlc"]
Nov 03 23:28:47.646878 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:47.646844    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e73b381_beec_43fe_ba3d_460efc605f8b.slice/crio-2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6 WatchSource:0}: Error finding container 2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6: Status 404 returned error can't find the container with id 2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6
Nov 03 23:28:47.767482 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.766677    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9" event={"ID":"59264120-38b0-4c6d-b3be-7d0340db0e34","Type":"ContainerStarted","Data":"b18d2e2539d79098695b1c0376635a481d466dc08a2db8b88703c87115d61af6"}
Nov 03 23:28:47.770937 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.770905    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7cjjg" event={"ID":"d2d422d3-fe51-4985-8577-4ddfcc24ba8c","Type":"ContainerStarted","Data":"64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e"}
Nov 03 23:28:47.773191 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:47.773129    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcvlc" event={"ID":"3e73b381-beec-43fe-ba3d-460efc605f8b","Type":"ContainerStarted","Data":"2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6"}
Nov 03 23:28:48.779874 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:48.778912    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/0.log"
Nov 03 23:28:48.779874 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:48.778955    2580 generic.go:356] "Generic (PLEG): container finished" podID="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" containerID="51c73c8823110f7cebf6bf8247c24b8d1b3f4870a150ab799632d7616c89dcc8" exitCode=255
Nov 03 23:28:48.779874 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:48.779307    2580 scope.go:122] "RemoveContainer" containerID="51c73c8823110f7cebf6bf8247c24b8d1b3f4870a150ab799632d7616c89dcc8"
Nov 03 23:28:48.779874 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:48.779606    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" event={"ID":"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb","Type":"ContainerDied","Data":"51c73c8823110f7cebf6bf8247c24b8d1b3f4870a150ab799632d7616c89dcc8"}
Nov 03 23:28:48.785060 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:48.784774    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7869h" event={"ID":"69d478d4-7d64-4a68-823e-29c6b40b4a94","Type":"ContainerStarted","Data":"ebdfd07d9283bb1e23b0f5a8471d3f2072f1fcfddcdb641d6006184613274af7"}
Nov 03 23:28:48.800764 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:48.800699    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9" podStartSLOduration=420.454058353 podStartE2EDuration="7m2.800684029s" totalImagesPullingTime="2.346625676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:47.785543379 +0000 UTC m=+35.018466070" watchObservedRunningTime="2026-11-03 23:28:48.800684029 +0000 UTC m=+36.033606722"
Nov 03 23:28:49.755774 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.754329    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7869h" podStartSLOduration=419.480836744 podStartE2EDuration="7m3.754308154s" totalImagesPullingTime="4.27347141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:48.82303277 +0000 UTC m=+36.055955462" watchObservedRunningTime="2026-11-03 23:28:49.754308154 +0000 UTC m=+36.987230846"
Nov 03 23:28:49.755774 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.754767    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dzv94"]
Nov 03 23:28:49.779068 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.779035    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dzv94"]
Nov 03 23:28:49.779231 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.779187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.786861 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.785211    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 23:28:49.786861 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.785379    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2bch\""
Nov 03 23:28:49.786861 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.785447    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 23:28:49.786861 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.785828    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ddq79-rh2q9"]
Nov 03 23:28:49.801402 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.801384    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ddq79-rh2q9"]
Nov 03 23:28:49.803462 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.801481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.806210 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.806192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.806373 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.806357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.806493 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.806477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75649c60-b6dd-4fe0-96f7-ff34c763de46-crio-socket\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.806597 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.806585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.806755 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.806726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.806897 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.806882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.807057 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.807043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2gn\" (UniqueName: \"kubernetes.io/projected/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-api-access-hb2gn\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.807202 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.807190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.807349 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.807336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75649c60-b6dd-4fe0-96f7-ff34c763de46-data-volume\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.817503 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.817480    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc" event={"ID":"3cb8c88a-1985-4ca8-a11f-7ab86cd87634","Type":"ContainerStarted","Data":"43969488b58c725d8d66e26a00d77105d360d9d9c988db90467f3b8ee67291c3"}
Nov 03 23:28:49.830636 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.830553    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:28:49.844274 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.843986    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/0.log"
Nov 03 23:28:49.844274 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.844026    2580 generic.go:356] "Generic (PLEG): container finished" podID="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" containerID="3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495" exitCode=255
Nov 03 23:28:49.845172 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.844818    2580 scope.go:122] "RemoveContainer" containerID="3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495"
Nov 03 23:28:49.845172 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.844854    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" event={"ID":"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb","Type":"ContainerDied","Data":"3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495"}
Nov 03 23:28:49.845172 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.844896    2580 scope.go:122] "RemoveContainer" containerID="51c73c8823110f7cebf6bf8247c24b8d1b3f4870a150ab799632d7616c89dcc8"
Nov 03 23:28:49.852719 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.852382    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc" podStartSLOduration=418.656372769 podStartE2EDuration="7m3.852367079s" totalImagesPullingTime="5.19599431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:49.851020562 +0000 UTC m=+37.083943262" watchObservedRunningTime="2026-11-03 23:28:49.852367079 +0000 UTC m=+37.085289771"
Nov 03 23:28:49.880831 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:49.879015    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6sdkj_openshift-console-operator(e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" podUID="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb"
Nov 03 23:28:49.880831 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.879563    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj" event={"ID":"13b0dc51-f5cc-4afd-821f-c3097be21e80","Type":"ContainerStarted","Data":"3239e2bd93fc77aa944e825b031ff9ad3e2af27dbc65fc0f50f3fc8a87c5de10"}
Nov 03 23:28:49.908080 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.907721    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj" podStartSLOduration=419.00260612 podStartE2EDuration="7m3.907700912s" totalImagesPullingTime="4.905094792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:49.906718243 +0000 UTC m=+37.139640935" watchObservedRunningTime="2026-11-03 23:28:49.907700912 +0000 UTC m=+37.140623605"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75649c60-b6dd-4fe0-96f7-ff34c763de46-data-volume\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75649c60-b6dd-4fe0-96f7-ff34c763de46-crio-socket\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.908902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2gn\" (UniqueName: \"kubernetes.io/projected/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-api-access-hb2gn\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.909455 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.909082    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75649c60-b6dd-4fe0-96f7-ff34c763de46-data-volume\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:49.909688    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:49.909787    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:50.409764843 +0000 UTC m=+37.642687531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.909979    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.910245    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:49.910302    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:49.910348    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:50.410332134 +0000 UTC m=+37.643254822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : configmap "kube-rbac-proxy" not found
Nov 03 23:28:49.911131 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.910498    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75649c60-b6dd-4fe0-96f7-ff34c763de46-crio-socket\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:49.911449 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.911428    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.934385 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.934356    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:49.942946 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:49.942921    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2gn\" (UniqueName: \"kubernetes.io/projected/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-api-access-hb2gn\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:50.169631 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.169510    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"]
Nov 03 23:28:50.179018 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.178821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:50.190798 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.189903    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"]
Nov 03 23:28:50.190798 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.189953    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"]
Nov 03 23:28:50.193516 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.192885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:50.211727 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.211699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kbjd\" (UniqueName: \"kubernetes.io/projected/c752b04e-c068-4c8c-94d3-405ff6ab27bb-kube-api-access-9kbjd\") pod \"migrator-66df5d496b-nls48\" (UID: \"c752b04e-c068-4c8c-94d3-405ff6ab27bb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:50.220043 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.220019    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"]
Nov 03 23:28:50.220169 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.220155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:50.220235 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.220182    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rg97z\""
Nov 03 23:28:50.221775 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.221754    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:50.221870 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.221829    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 23:28:50.313015 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.312390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbjd\" (UniqueName: \"kubernetes.io/projected/c752b04e-c068-4c8c-94d3-405ff6ab27bb-kube-api-access-9kbjd\") pod \"migrator-66df5d496b-nls48\" (UID: \"c752b04e-c068-4c8c-94d3-405ff6ab27bb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:50.313015 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.312568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwsc\" (UniqueName: \"kubernetes.io/projected/9763223d-ebce-4993-9932-d443787cd21c-kube-api-access-lkwsc\") pod \"migrator-66df5d496b-bs5gg\" (UID: \"9763223d-ebce-4993-9932-d443787cd21c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:50.326559 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.326529    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbjd\" (UniqueName: \"kubernetes.io/projected/c752b04e-c068-4c8c-94d3-405ff6ab27bb-kube-api-access-9kbjd\") pod \"migrator-66df5d496b-nls48\" (UID: \"c752b04e-c068-4c8c-94d3-405ff6ab27bb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.413116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwsc\" (UniqueName: \"kubernetes.io/projected/9763223d-ebce-4993-9932-d443787cd21c-kube-api-access-lkwsc\") pod \"migrator-66df5d496b-bs5gg\" (UID: \"9763223d-ebce-4993-9932-d443787cd21c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.413652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.413754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:50.413800    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:50.413859    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.413842134 +0000 UTC m=+38.646764825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:50.413904    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 23:28:50.413994 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:50.413961    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:51.413948074 +0000 UTC m=+38.646870747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : configmap "kube-rbac-proxy" not found
Nov 03 23:28:50.424976 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.424923    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwsc\" (UniqueName: \"kubernetes.io/projected/9763223d-ebce-4993-9932-d443787cd21c-kube-api-access-lkwsc\") pod \"migrator-66df5d496b-bs5gg\" (UID: \"9763223d-ebce-4993-9932-d443787cd21c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:50.529306 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.529275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:50.555553 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.555236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:50.634571 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.634462    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ddq79-rh2q9"]
Nov 03 23:28:50.843684 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.843652    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"]
Nov 03 23:28:50.846435 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:50.846408    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc752b04e_c068_4c8c_94d3_405ff6ab27bb.slice/crio-f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66 WatchSource:0}: Error finding container f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66: Status 404 returned error can't find the container with id f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66
Nov 03 23:28:50.884702 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.884665    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48" event={"ID":"c752b04e-c068-4c8c-94d3-405ff6ab27bb","Type":"ContainerStarted","Data":"f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66"}
Nov 03 23:28:50.886701 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.886678    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ddq79-rh2q9" event={"ID":"4a23c360-8e0e-4985-ba27-a6631df95e11","Type":"ContainerStarted","Data":"6d9b50efdd5bbed2f29bf7de9d13d454a607b89fee494995e709ace5a37181ee"}
Nov 03 23:28:50.888931 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.888903    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6mthf" event={"ID":"3b107666-eaaa-4d08-839b-d3e91dd5868b","Type":"ContainerStarted","Data":"1e3cc419d195652ad950c0723bf52e400f22c84656b0e909cb6b37f1dd5c58ed"}
Nov 03 23:28:50.890945 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.890901    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:28:50.891352 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.891333    2580 scope.go:122] "RemoveContainer" containerID="3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495"
Nov 03 23:28:50.891532 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:50.891493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6sdkj_openshift-console-operator(e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" podUID="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb"
Nov 03 23:28:50.908396 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.908369    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"]
Nov 03 23:28:50.952957 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:50.952901    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6mthf" podStartSLOduration=2.010203335 podStartE2EDuration="7.9528838s" totalImagesPullingTime="5.942680465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:50.913095454 +0000 UTC m=+38.146018146" watchObservedRunningTime="2026-11-03 23:28:50.9528838 +0000 UTC m=+38.185806494"
Nov 03 23:28:51.008092 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.008062    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d5bgc"]
Nov 03 23:28:51.023094 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.023041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.025758 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.025030    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d5bgc"]
Nov 03 23:28:51.027580 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.027555    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 23:28:51.027891 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.027863    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m6pk6\""
Nov 03 23:28:51.027972 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.027945    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 23:28:51.031898 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.031865    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 23:28:51.122113 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.121977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-cabundle\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.122256 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.122127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljwhm\" (UniqueName: \"kubernetes.io/projected/ebc10a97-edbf-42d7-bfd5-caa1d573329e-kube-api-access-ljwhm\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.122256 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.122168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-key\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.127377 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.127347    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 23:28:51.223771 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.223700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljwhm\" (UniqueName: \"kubernetes.io/projected/ebc10a97-edbf-42d7-bfd5-caa1d573329e-kube-api-access-ljwhm\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.223958 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.223786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-key\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.224045 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.223960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-cabundle\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.224649 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.224616    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-cabundle\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.227572 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.227540    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-key\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.236702 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.236679    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljwhm\" (UniqueName: \"kubernetes.io/projected/ebc10a97-edbf-42d7-bfd5-caa1d573329e-kube-api-access-ljwhm\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.349526 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.349499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:51.426525 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:51.426641 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:51.426641 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:51.426700 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:51.426767 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:51.426821 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:51.426821 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.426816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:51.426926 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.426912    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 23:28:51.426985 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.426975    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:53.426957447 +0000 UTC m=+40.659880122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : configmap "kube-rbac-proxy" not found
Nov 03 23:28:51.427372 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427357    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 23:28:51.427416 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427405    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.42739134 +0000 UTC m=+46.660314009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : secret "router-metrics-certs-default" not found
Nov 03 23:28:51.427472 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427462    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle podName:698112a3-0157-43a7-9ea6-30e554d8be51 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.427453263 +0000 UTC m=+46.660375939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle") pod "router-default-5bb58b79c9-bksx8" (UID: "698112a3-0157-43a7-9ea6-30e554d8be51") : configmap references non-existent config key: service-ca.crt
Nov 03 23:28:51.427532 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427522    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:51.427573 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427552    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.427543693 +0000 UTC m=+46.660466366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:51.427621 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427605    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:51.427660 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427634    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls podName:4741d3bf-14a8-4118-a61e-32dd236755dd nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.427624762 +0000 UTC m=+46.660547434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ngdxl" (UID: "4741d3bf-14a8-4118-a61e-32dd236755dd") : secret "cluster-monitoring-operator-tls" not found
Nov 03 23:28:51.427701 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427684    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:51.427730 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:53.427711629 +0000 UTC m=+40.660634304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:51.427807 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427795    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:51.427851 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427811    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:51.427851 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.427845    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.4278359 +0000 UTC m=+46.660758574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.527398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.527496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.527578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527697    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527708    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527766    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.527734357 +0000 UTC m=+46.760657040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527861    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527919    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert podName:dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.527906162 +0000 UTC m=+46.760828849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cs8jk" (UID: "dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d") : secret "networking-console-plugin-cert" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527861    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:51.528025 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:51.527957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.527947034 +0000 UTC m=+46.760869718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:51.681594 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.681557    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d5bgc"]
Nov 03 23:28:51.686343 ip-10-0-138-239 kubenswrapper[2580]: W1103 23:28:51.686306    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc10a97_edbf_42d7_bfd5_caa1d573329e.slice/crio-a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c WatchSource:0}: Error finding container a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c: Status 404 returned error can't find the container with id a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c
Nov 03 23:28:51.901182 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.901061    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7cjjg" event={"ID":"d2d422d3-fe51-4985-8577-4ddfcc24ba8c","Type":"ContainerStarted","Data":"5ac8c9bf0b50dce911c5b4ea59e1fd3c462b0a300662c9a4cb812fd424cd7dd2"}
Nov 03 23:28:51.901645 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.901246    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:51.904136 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.904094    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg" event={"ID":"9763223d-ebce-4993-9932-d443787cd21c","Type":"ContainerStarted","Data":"d8d7c518f82d0925be2a712a922f4ccfa02b542ded12d3f3d0857aab597ee067"}
Nov 03 23:28:51.906463 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.906438    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ddq79-rh2q9" event={"ID":"4a23c360-8e0e-4985-ba27-a6631df95e11","Type":"ContainerStarted","Data":"2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af"}
Nov 03 23:28:51.908511 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.908488    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v" event={"ID":"17673e8b-0dec-48d4-8843-65147e22372a","Type":"ContainerStarted","Data":"797268fa4acce7e8a045727caf1f6376ad60e245b821866f6166b9ec9eb9c7be"}
Nov 03 23:28:51.913241 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.913216    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc" event={"ID":"ebc10a97-edbf-42d7-bfd5-caa1d573329e","Type":"ContainerStarted","Data":"df72bc9a2108a2784b7560ac3781d68c9a2a79599edff7903653629b7c6c3b8a"}
Nov 03 23:28:51.913375 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.913312    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc" event={"ID":"ebc10a97-edbf-42d7-bfd5-caa1d573329e","Type":"ContainerStarted","Data":"a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c"}
Nov 03 23:28:51.924078 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.924036    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7cjjg" podStartSLOduration=35.201802585 podStartE2EDuration="38.924023878s" totalImagesPullingTime="3.722221293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:51.922459499 +0000 UTC m=+39.155382191" watchObservedRunningTime="2026-11-03 23:28:51.924023878 +0000 UTC m=+39.156946569"
Nov 03 23:28:51.948052 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.946903    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v" podStartSLOduration=413.450309233 podStartE2EDuration="6m59.946885459s" totalImagesPullingTime="6.496576226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:51.945810186 +0000 UTC m=+39.178732879" watchObservedRunningTime="2026-11-03 23:28:51.946885459 +0000 UTC m=+39.179808177"
Nov 03 23:28:51.963711 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.963666    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc" podStartSLOduration=1.963651459 podStartE2EDuration="1.963651459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:51.961379039 +0000 UTC m=+39.194301731" watchObservedRunningTime="2026-11-03 23:28:51.963651459 +0000 UTC m=+39.196574151"
Nov 03 23:28:51.979474 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:51.979435    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ddq79-rh2q9" podStartSLOduration=2.979420955 podStartE2EDuration="2.979420955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:51.978232615 +0000 UTC m=+39.211155307" watchObservedRunningTime="2026-11-03 23:28:51.979420955 +0000 UTC m=+39.212343646"
Nov 03 23:28:53.218381 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.218341    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c99qk_c4738a53-824f-4a8b-ab86-b160b9f14c0e/dns-node-resolver/0.log"
Nov 03 23:28:53.450131 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.450089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:53.450314 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.450189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:53.450314 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:53.450256    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:53.450432 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:53.450319    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.450299106 +0000 UTC m=+44.683221796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:53.450432 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:53.450341    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 23:28:53.450432 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:53.450376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.450365737 +0000 UTC m=+44.683288420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : configmap "kube-rbac-proxy" not found
Nov 03 23:28:53.921005 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.920912    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48" event={"ID":"c752b04e-c068-4c8c-94d3-405ff6ab27bb","Type":"ContainerStarted","Data":"3d48e4388df3d1c4f53f3ffa3b8471824b461b9a715a5f590632a928c4d78edf"}
Nov 03 23:28:53.922603 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.922579    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg" event={"ID":"9763223d-ebce-4993-9932-d443787cd21c","Type":"ContainerStarted","Data":"bfc36916fc3c2a2f1d8086b304e7b272c567c508ebe6730f694d2790cb75bde1"}
Nov 03 23:28:53.939709 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.939668    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48" podStartSLOduration=1.941752899 podStartE2EDuration="3.939654484s" totalImagesPullingTime="1.997901585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:53.938778409 +0000 UTC m=+41.171701100" watchObservedRunningTime="2026-11-03 23:28:53.939654484 +0000 UTC m=+41.172577175"
Nov 03 23:28:53.977221 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.977191    2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:53.977403 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.977261    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:53.978091 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:53.978062    2580 scope.go:122] "RemoveContainer" containerID="3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495"
Nov 03 23:28:53.978412 ip-10-0-138-239 kubenswrapper[2580]: E1103 23:28:53.978387    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6sdkj_openshift-console-operator(e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" podUID="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb"
Nov 03 23:28:54.222042 ip-10-0-138-239 kubenswrapper[2580]: I1103 23:28:54.222013    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdxlq_cd9bbc11-d972-4662-bb0d-b1d8744673c9/node-ca/0.log"
Nov 03 23:28:54.795687 ip-10-0-138-239 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 23:28:54.816210 ip-10-0-138-239 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 23:28:54.816433 ip-10-0-138-239 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 23:28:54.816492 ip-10-0-138-239 systemd[1]: kubelet.service: Consumed 3.200s CPU time.
Nov 03 23:28:54.828245 ip-10-0-138-239 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 23:28:54.929331 ip-10-0-138-239 kubenswrapper[5074]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:54.929331 ip-10-0-138-239 kubenswrapper[5074]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 23:28:54.929331 ip-10-0-138-239 kubenswrapper[5074]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:54.929331 ip-10-0-138-239 kubenswrapper[5074]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:54.932423 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932336    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.932423 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932420    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.932423 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932425    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.932423 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932429    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.932423 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932432    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932435    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932438    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932441    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932443    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932446    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932449    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932452    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932455    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932458    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932461    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932464    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932466    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932469    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932472    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932474    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932481    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932484    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932487    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932490    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.932619 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932493    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932496    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932498    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932501    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932504    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932506    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932509    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932511    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932514    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932517    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932520    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932522    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932525    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932527    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932530    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932532    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932535    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932537    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932540    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932542    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.933128 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932546    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932549    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932551    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932554    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932556    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932559    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932563    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932566    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932569    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932571    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932574    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932576    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932579    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932582    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932585    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932588    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932591    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932594    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932597    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932599    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.933606 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932602    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932604    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932607    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932610    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932612    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932615    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932619    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932624    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932628    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932631    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932634    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932637    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932640    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932642    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932645    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932649    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932652    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932656    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932659    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.934111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932662    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932665    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932668    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932671    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932674    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932677    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932680    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932683    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932685    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932688    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932691    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932694    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932697    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932699    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932702    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932705    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932709    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932711    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932714    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.934563 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932716    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932719    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932721    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.932724    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933473    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933480    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933483    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933486    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933489    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933492    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933496    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933500    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933503    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933505    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933509    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933512    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933515    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933517    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933520    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933523    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.935062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933526    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933528    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933531    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933534    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933536    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933539    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933542    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933545    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933548    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933551    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933553    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933556    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933559    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933561    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933564    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933566    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933569    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933571    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933574    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.935551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933577    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933579    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933582    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933584    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933587    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933589    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933592    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933594    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933598    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933600    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933603    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933606    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933610    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933612    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933615    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933618    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933621    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933624    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933627    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933630    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.936073 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933632    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933634    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933637    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933639    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933642    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933644    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933647    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933649    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933652    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933656    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933659    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933662    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933664    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933667    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933670    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933672    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933675    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933678    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933680    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.936556 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933682    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933685    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933689    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933692    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933694    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933697    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933700    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933702    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933705    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933707    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933710    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933713    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933715    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933718    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933721    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933723    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933726    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933729    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933731    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933733    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.937040 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933752    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933756    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933760    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933764    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933768    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933773    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933777    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933781    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933784    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933786    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933789    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.933794    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934283    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934288    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934292    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934295    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934299    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934302    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934304    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.937559 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934307    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934310    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934312    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934315    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934318    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934320    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934323    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934325    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934328    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934331    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934333    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934336    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934338    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934341    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934343    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934346    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934348    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934351    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934354    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934356    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.938065 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934359    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934361    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934364    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934366    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934369    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934372    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934374    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934377    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934381    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934384    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934387    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934390    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934392    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934394    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934397    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934400    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934402    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934404    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934407    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934410    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.938551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934412    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934414    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934417    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934420    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934423    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934428    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934432    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934436    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934440    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934443    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934445    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934448    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934450    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934453    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934455    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934458    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934460    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934463    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934466    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.939037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934469    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934472    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934477    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934480    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934483    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934485    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934488    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934495    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934498    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934500    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934503    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934505    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934508    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934510    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934513    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934515    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934518    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934521    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934523    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.939482 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934526    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934528    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934531    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934534    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934536    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934539    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934541    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934544    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934546    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934550    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934552    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934555    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934558    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934560    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934563    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934565    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934569    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934572    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934574    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934577    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.939948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.934581    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934679    5074 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934695    5074 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934704    5074 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934711    5074 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934715    5074 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934719    5074 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934723    5074 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934727    5074 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934731    5074 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934734    5074 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934750    5074 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934755    5074 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934760    5074 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934764    5074 flags.go:64] FLAG: --cgroup-root=""
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934769    5074 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934774    5074 flags.go:64] FLAG: --client-ca-file=""
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934777    5074 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934780    5074 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934786    5074 flags.go:64] FLAG: --cluster-domain=""
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934788    5074 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934792    5074 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934795    5074 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934798    5074 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 23:28:54.940432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934802    5074 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934806    5074 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934809    5074 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934812    5074 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934815    5074 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934819    5074 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934823    5074 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934827    5074 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934832    5074 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934841    5074 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934844    5074 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934847    5074 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934850    5074 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934853    5074 flags.go:64] FLAG: --enable-server="true"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934856    5074 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934861    5074 flags.go:64] FLAG: --event-burst="100"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934864    5074 flags.go:64] FLAG: --event-qps="50"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934867    5074 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934870    5074 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934873    5074 flags.go:64] FLAG: --eviction-hard=""
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934877    5074 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934880    5074 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934884    5074 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934887    5074 flags.go:64] FLAG: --eviction-soft=""
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934890    5074 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 23:28:54.941028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934893    5074 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934896    5074 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934899    5074 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934902    5074 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934905    5074 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934908    5074 flags.go:64] FLAG: --feature-gates=""
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934913    5074 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934916    5074 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934919    5074 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934922    5074 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934925    5074 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934929    5074 flags.go:64] FLAG: --help="false"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934932    5074 flags.go:64] FLAG: --hostname-override="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934936    5074 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934939    5074 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934944    5074 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934947    5074 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934951    5074 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934954    5074 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934957    5074 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934960    5074 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934963    5074 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934966    5074 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 23:28:54.941619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934969    5074 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934974    5074 flags.go:64] FLAG: --kube-reserved=""
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934977    5074 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934979    5074 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934982    5074 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934985    5074 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934988    5074 flags.go:64] FLAG: --lock-file=""
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934991    5074 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934994    5074 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.934997    5074 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935002    5074 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935005    5074 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935008    5074 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935011    5074 flags.go:64] FLAG: --logging-format="text"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935014    5074 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935018    5074 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935021    5074 flags.go:64] FLAG: --manifest-url=""
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935024    5074 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935029    5074 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935032    5074 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935035    5074 flags.go:64] FLAG: --max-pods="110"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935038    5074 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935041    5074 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935045    5074 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935049    5074 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 23:28:54.942214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935052    5074 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935055    5074 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935058    5074 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935066    5074 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935069    5074 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935072    5074 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935076    5074 flags.go:64] FLAG: --pod-cidr=""
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935079    5074 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935083    5074 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935087    5074 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935090    5074 flags.go:64] FLAG: --port="10250"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935093    5074 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935096    5074 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0526ede6620beaf57"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935099    5074 flags.go:64] FLAG: --qos-reserved=""
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935102    5074 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935105    5074 flags.go:64] FLAG: --register-node="true"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935108    5074 flags.go:64] FLAG: --register-with-taints=""
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935112    5074 flags.go:64] FLAG: --registry-burst="10"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935115    5074 flags.go:64] FLAG: --registry-qps="5"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935118    5074 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935121    5074 flags.go:64] FLAG: --reserved-memory=""
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935125    5074 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935128    5074 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935131    5074 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935134    5074 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935137    5074 flags.go:64] FLAG: --runonce="false"
Nov 03 23:28:54.942904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935140    5074 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935144    5074 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935147    5074 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935150    5074 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935153    5074 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935158    5074 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935161    5074 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935164    5074 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935167    5074 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935170    5074 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935173    5074 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935176    5074 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935179    5074 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935182    5074 flags.go:64] FLAG: --system-cgroups=""
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935186    5074 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935191    5074 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935194    5074 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935197    5074 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935201    5074 flags.go:64] FLAG: --tls-min-version=""
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935204    5074 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935207    5074 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935210    5074 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935213    5074 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935216    5074 flags.go:64] FLAG: --v="2"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935220    5074 flags.go:64] FLAG: --version="false"
Nov 03 23:28:54.943531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935224    5074 flags.go:64] FLAG: --vmodule=""
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935228    5074 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935232    5074 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935364    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935368    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935372    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935375    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935377    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935380    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935383    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935386    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935388    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935393    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935396    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935399    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935401    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935405    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935409    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935412    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.944219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935417    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935420    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935425    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935429    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935432    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935434    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935437    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935440    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935443    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935446    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935449    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935451    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935454    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935456    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935459    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935462    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935464    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935467    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935470    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935472    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.944716 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935475    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935478    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935480    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935483    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935486    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935490    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935492    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935499    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935501    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935504    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935506    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935509    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935512    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935515    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935518    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935521    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935524    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935526    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935529    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935531    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.945250 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935534    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935536    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935539    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935542    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935544    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935547    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935549    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935552    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935555    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935557    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935560    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935563    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935565    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935568    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935570    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935573    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935576    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935579    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935581    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.946048 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935584    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935587    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935589    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935592    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935595    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935597    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935600    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935604    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935609    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935613    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935616    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935620    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935625    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935629    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935634    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935637    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935639    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935642    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935645    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.946688 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935647    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935650    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935652    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935655    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935658    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935660    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935663    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935665    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935667    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935670    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935673    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935675    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.935702    5074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935752    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935758    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935764    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.947182 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935769    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935773    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935775    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935778    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935783    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935785    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935788    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935791    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935794    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935796    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935799    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935801    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935804    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935807    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935809    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935812    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935815    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935818    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935820    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935823    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.947597 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935825    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935828    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935830    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935833    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935835    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935838    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935840    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935843    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935845    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935848    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935850    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935853    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935856    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935859    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935861    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935864    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935869    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935871    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935874    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.948106 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935876    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935879    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935881    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935884    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935886    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935889    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935892    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935894    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935897    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935899    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935902    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935905    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935907    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935909    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935912    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935915    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935918    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935920    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935923    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.948673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935925    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935928    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935931    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935933    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935935    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935938    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935940    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935944    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935947    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935949    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935953    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935955    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935958    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935961    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935963    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935967    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935970    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935972    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935975    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.949391 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935978    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935981    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935984    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935986    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935989    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935992    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935994    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935997    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.935999    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936003    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936007    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936010    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936014    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936019    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936022    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936025    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936028    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936030    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936033    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.950011 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936036    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936038    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936041    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936044    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936048    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936050    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.936053    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.939606    5074 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.939623    5074 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.939640    5074 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.939645    5074 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939699    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939705    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939709    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939711    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939715    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939718    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939721    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939723    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939726    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.950494 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939729    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939731    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939734    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939751    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939756    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939761    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939766    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939769    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939772    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939775    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939778    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939781    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939783    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939785    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939789    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939791    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939794    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939796    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939799    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939802    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.951008 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939805    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939808    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939811    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939813    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939816    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939819    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939822    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939824    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939827    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939829    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939832    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939835    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939838    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939840    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939843    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939845    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939848    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939852    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939857    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.951489 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939860    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939863    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939867    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939869    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939872    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939875    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939877    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939880    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939882    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939885    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939888    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939890    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939893    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939895    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939899    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939901    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939904    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939907    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939909    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939912    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.951948 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939914    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939917    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939920    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939922    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939925    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939927    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939930    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939933    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939935    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939938    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939940    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939942    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939945    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939949    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939952    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939955    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939957    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939960    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939963    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939965    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.952432 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939968    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939970    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939973    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939975    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939978    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939980    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939982    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939985    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939988    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939990    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939993    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.939997    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940001    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940015    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940020    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940024    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940037    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.953051 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940042    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.940071    5074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940207    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940211    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940215    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940218    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940220    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940223    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940225    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940228    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940231    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940233    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940236    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940238    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940241    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940243    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.953453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940246    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940248    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940251    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940254    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940256    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940259    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940261    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940264    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940267    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940270    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940273    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940276    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940278    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940281    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940283    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940286    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940289    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940291    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940294    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940296    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.953858 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940299    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940302    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940305    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940307    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940310    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940312    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940315    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940317    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940320    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940322    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940324    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940327    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940329    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940332    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940334    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940337    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940342    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940344    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940347    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940350    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.954470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940353    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940356    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940358    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940361    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940364    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940366    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940370    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940374    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940377    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940380    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940382    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940385    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940388    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940391    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940393    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940396    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940398    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940401    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940404    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.955002 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940406    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940409    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940412    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940414    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940418    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940422    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940425    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940427    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940430    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940434    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940437    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940439    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940442    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940445    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940449    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940452    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940454    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940457    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940460    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.955459 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940462    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940465    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940467    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940470    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940472    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940475    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940478    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940481    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940483    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940486    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940488    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940491    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940493    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940496    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.940519    5074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940549    5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:54.955971 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940553    5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940556    5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940559    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940562    5074 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940565    5074 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940567    5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940570    5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940574    5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940576    5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940579    5074 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940582    5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940584    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940587    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940591    5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940593    5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940596    5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940599    5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940601    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940604    5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940606    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:54.956400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940609    5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940611    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940614    5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940617    5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940619    5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940622    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940625    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940627    5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940630    5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940632    5074 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940635    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940638    5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940640    5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940645    5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940649    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940652    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940655    5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940658    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940661    5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:54.957050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940666    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940668    5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940671    5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940674    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940676    5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940679    5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940682    5074 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940685    5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940687    5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940690    5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940693    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940697    5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940700    5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940704    5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940707    5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940710    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940712    5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940715    5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940718    5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:54.957515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940720    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940723    5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940726    5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940729    5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940731    5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940734    5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940752    5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940755    5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940758    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940760    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940763    5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940765    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940768    5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940771    5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940774    5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940777    5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940780    5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940782    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940785    5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:54.958049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940788    5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940791    5074 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940793    5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940796    5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940798    5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940801    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940803    5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940806    5074 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940808    5074 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940811    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940814    5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940817    5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940819    5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940822    5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940824    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940827    5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940830    5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940832    5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940835    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940837    5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:54.958509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940840    5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940842    5074 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940845    5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940847    5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940850    5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940852    5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940856    5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:54.940860    5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.940909    5074 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.941856    5074 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.941910    5074 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.942118    5074 server.go:1014] "Starting client certificate rotation"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.942252    5074 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.942306    5074 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 23:23:13 +0000 UTC" deadline="2028-07-08 01:09:07.776530044 +0000 UTC"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.942333    5074 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14689h40m12.834199266s"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.942572    5074 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:28:54.959058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.943548    5074 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:28:54.959441 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.945110    5074 log.go:25] "Validated CRI v1 runtime API"
Nov 03 23:28:54.959441 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.948510    5074 log.go:25] "Validated CRI v1 image API"
Nov 03 23:28:54.959441 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.949300    5074 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 23:28:54.959441 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.951365    5074 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ba5eac8c-fe68-40b6-84d6-fbad63022036:/dev/nvme0n1p3 bbb731a6-bffc-4465-9458-b86bb2c92502:/dev/nvme0n1p4]
Nov 03 23:28:54.959574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.951388    5074 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d9b50efdd5bbed2f29bf7de9d13d454a607b89fee494995e709ace5a37181ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d9b50efdd5bbed2f29bf7de9d13d454a607b89fee494995e709ace5a37181ee/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8d7c518f82d0925be2a712a922f4ccfa02b542ded12d3f3d0857aab597ee067/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8d7c518f82d0925be2a712a922f4ccfa02b542ded12d3f3d0857aab597ee067/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~projected/kube-api-access-m9gr2:{mountpoint:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~projected/kube-api-access-m9gr2 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1244c649-9fe7-40c5-9b55-443a2529c710/volumes/kubernetes.io~projected/kube-api-access-62llt:{mountpoint:/var/lib/kubelet/pods/1244c649-9fe7-40c5-9b55-443a2529c710/volumes/kubernetes.io~projected/kube-api-access-62llt major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13b0dc51-f5cc-4afd-821f-c3097be21e80/volumes/kubernetes.io~projected/kube-api-access-c59s2:{mountpoint:/var/lib/kubelet/pods/13b0dc51-f5cc-4afd-821f-c3097be21e80/volumes/kubernetes.io~projected/kube-api-access-c59s2 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13b0dc51-f5cc-4afd-821f-c3097be21e80/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/13b0dc51-f5cc-4afd-821f-c3097be21e80/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17673e8b-0dec-48d4-8843-65147e22372a/volumes/kubernetes.io~projected/kube-api-access-vw4zq:{mountpoint:/var/lib/kubelet/pods/17673e8b-0dec-48d4-8843-65147e22372a/volumes/kubernetes.io~projected/kube-api-access-vw4zq major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~projected/kube-api-access-smznp:{mountpoint:/var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~projected/kube-api-access-smznp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b107666-eaaa-4d08-839b-d3e91dd5868b/volumes/kubernetes.io~projected/kube-api-access-g8vfw:{mountpoint:/var/lib/kubelet/pods/3b107666-eaaa-4d08-839b-d3e91dd5868b/volumes/kubernetes.io~projected/kube-api-access-g8vfw major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b107666-eaaa-4d08-839b-d3e91dd5868b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3b107666-eaaa-4d08-839b-d3e91dd5868b/volumes/kubernetes.io~secret/cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb8c88a-1985-4ca8-a11f-7ab86cd87634/volumes/kubernetes.io~projected/kube-api-access-gc8lk:{mountpoint:/var/lib/kubelet/pods/3cb8c88a-1985-4ca8-a11f-7ab86cd87634/volumes/kubernetes.io~projected/kube-api-access-gc8lk major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb8c88a-1985-4ca8-a11f-7ab86cd87634/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3cb8c88a-1985-4ca8-a11f-7ab86cd87634/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e73b381-beec-43fe-ba3d-460efc605f8b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3e73b381-beec-43fe-ba3d-460efc605f8b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40135659-f74d-4307-8bb3-574411040f5a/volumes/kubernetes.io~projected/kube-api-access-khhwq:{mountpoint:/var/lib/kubelet/pods/40135659-f74d-4307-8bb3-574411040f5a/volumes/kubernetes.io~projected/kube-api-access-khhwq major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4741d3bf-14a8-4118-a61e-32dd236755dd/volumes/kubernetes.io~projected/kube-api-access-fkdbc:{mountpoint:/var/lib/kubelet/pods/4741d3bf-14a8-4118-a61e-32dd236755dd/volumes/kubernetes.io~projected/kube-api-access-fkdbc major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47845d6a-a330-49f4-be05-d3a8aeb677dc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/47845d6a-a330-49f4-be05-d3a8aeb677dc/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a23c360-8e0e-4985-ba27-a6631df95e11/volumes/kubernetes.io~projected/kube-api-access-vbhzz:{mountpoint:/var/lib/kubelet/pods/4a23c360-8e0e-4985-ba27-a6631df95e11/volumes/kubernetes.io~projected/kube-api-access-vbhzz major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5032904c-5a18-4f28-be3b-a02375b14113/volumes/kubernetes.io~projected/kube-api-access-wsv66:{mountpoint:/var/lib/kubelet/pods/5032904c-5a18-4f28-be3b-a02375b14113/volumes/kubernetes.io~projected/kube-api-access-wsv66 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59264120-38b0-4c6d-b3be-7d0340db0e34/volumes/kubernetes.io~projected/kube-api-access-bcmx5:{mountpoint:/var/lib/kubelet/pods/59264120-38b0-4c6d-b3be-7d0340db0e34/volumes/kubernetes.io~projected/kube-api-access-bcmx5 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volumes/kubernetes.io~projected/kube-api-access-bvpvn:{mountpoint:/var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volumes/kubernetes.io~projected/kube-api-access-bvpvn major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63593602-b03b-47c5-9d74-a95ff2abfddb/volumes/kubernetes.io~projected/kube-api-access-jcl2v:{mountpoint:/var/lib/kubelet/pods/63593602-b03b-47c5-9d74-a95ff2abfddb/volumes/kubernetes.io~projected/kube-api-access-jcl2v major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~projected/kube-api-access-9twwh:{mountpoint:/var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~projected/kube-api-access-9twwh major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69d478d4-7d64-4a68-823e-29c6b40b4a94/volumes/kubernetes.io~projected/kube-api-access-kp6ff:{mountpoint:/var/lib/kubelet/pods/69d478d4-7d64-4a68-823e-29c6b40b4a94/volumes/kubernetes.io~projected/kube-api-access-kp6ff major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69d478d4-7d64-4a68-823e-29c6b40b4a94/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/69d478d4-7d64-4a68-823e-29c6b40b4a94/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75649c60-b6dd-4fe0-96f7-ff34c763de46/volumes/kubernetes.io~projected/kube-api-access-hb2gn:{mountpoint:/var/lib/kubelet/pods/75649c60-b6dd-4fe0-96f7-ff34c763de46/volumes/kubernetes.io~projected/kube-api-access-hb2gn major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9763223d-ebce-4993-9932-d443787cd21c/volumes/kubernetes.io~projected/kube-api-access-lkwsc:{mountpoint:/var/lib/kubelet/pods/9763223d-ebce-4993-9932-d443787cd21c/volumes/kubernetes.io~projected/kube-api-access-lkwsc major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1e7a5aa-87e3-4afd-ae59-801a559622f6/volumes/kubernetes.io~projected/kube-api-access-gbrrf:{mountpoint:/var/lib/kubelet/pods/a1e7a5aa-87e3-4afd-ae59-801a559622f6/volumes/kubernetes.io~projected/kube-api-access-gbrrf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1f0dd5d-1903-4b2a-830c-05e93d384fda/volumes/kubernetes.io~projected/kube-api-access-k6vm4:{mountpoint:/var/lib/kubelet/pods/a1f0dd5d-1903-4b2a-830c-05e93d384fda/volumes/kubernetes.io~projected/kube-api-access-k6vm4 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32ab19d-6150-424c-8ce9-d3c8fcf7f967/volumes/kubernetes.io~projected/kube-api-access-mm94g:{mountpoint:/var/lib/kubelet/pods/a32ab19d-6150-424c-8ce9-d3c8fcf7f967/volumes/kubernetes.io~projected/kube-api-access-mm94g major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/volumes/kubernetes.io~projected/kube-api-access-scd5g:{mountpoint:/var/lib/kubelet/pods/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/volumes/kubernetes.io~projected/kube-api-access-scd5g major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbbb34c1-9e43-4640-82ae-660e1e365930/volumes/kubernetes.io~projected/kube-api-access-6kmv7:{mountpoint:/var/lib/kubelet/pods/bbbb34c1-9e43-4640-82ae-660e1e365930/volumes/kubernetes.io~projected/kube-api-access-6kmv7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4738a53-824f-4a8b-ab86-b160b9f14c0e/volumes/kubernetes.io~projected/kube-api-access-vdjr2:{mountpoint:/var/lib/kubelet/pods/c4738a53-824f-4a8b-ab86-b160b9f14c0e/volumes/kubernetes.io~projected/kube-api-access-vdjr2 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c752b04e-c068-4c8c-94d3-405ff6ab27bb/volumes/kubernetes.io~projected/kube-api-access-9kbjd:{mountpoint:/var/lib/kubelet/pods/c752b04e-c068-4c8c-94d3-405ff6ab27bb/volumes/kubernetes.io~projected/kube-api-access-9kbjd major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd9bbc11-d972-4662-bb0d-b1d8744673c9/volumes/kubernetes.io~projected/kube-api-access-98v7q:{mountpoint:/var/lib/kubelet/pods/cd9bbc11-d972-4662-bb0d-b1d8744673c9/volumes/kubernetes.io~projected/kube-api-access-98v7q major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2d422d3-fe51-4985-8577-4ddfcc24ba8c/volumes/kubernetes.io~projected/kube-api-access-8662p:{mountpoint:/var/lib/kubelet/pods/d2d422d3-fe51-4985-8577-4ddfcc24ba8c/volumes/kubernetes.io~projected/kube-api-access-8662p major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/volumes/kubernetes.io~projected/kube-api-access-g86bg:{mountpoint:/var/lib/kubelet/pods/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/volumes/kubernetes.io~projected/kube-api-access-g86bg major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebc10a97-edbf-42d7-bfd5-caa1d573329e/volumes/kubernetes.io~projected/kube-api-access-ljwhm:{mountpoint:/var/lib/kubelet/pods/ebc10a97-edbf-42d7-bfd5-caa1d573329e/volumes/kubernetes.io~projected/kube-api-access-ljwhm major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebc10a97-edbf-42d7-bfd5-caa1d573329e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ebc10a97-edbf-42d7-bfd5-caa1d573329e/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~projected/kube-api-access-425hw:{mountpoint:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~projected/kube-api-access-425hw major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcd47e4a-ca1b-4557-82d0-200bdb384976/volumes/kubernetes.io~projected/kube-api-access-rcr4z:{mountpoint:/var/lib/kubelet/pods/fcd47e4a-ca1b-4557-82d0-200bdb384976/volumes/kubernetes.io~projected/kube-api-access-rcr4z major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/336ee2c18e6096ecdeef18a9aeffaebc9cd323c84baa0ebb4594d5f4371840fd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bae55b2798ac0bab33a8a022bbc1fcedc7b26b896180fda0f5fb1549cffd4ef4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5319825c8d333bb81e7f397df2725f7f51fdb8ae3b51fd289a87434ec5ade6fb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/c5345866a1a55511207110e8b3e759ed9666db855580f1fb01a7a75e6239f040/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2effef62e20b8c7d13b88d25f9717c1ceab1863a2122f6391c47ad6a57cb6739/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5d00352915f234c5e95cd7a1713ca1ba6ffc11fd24d9da81f4b5444a4f04183a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7f07b6895f4004eb2838cee0489d68f86f32532603c4002bce817a365c54decd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cdaae0e2c1603027091b6fa63785ebbcabfb0e8b71e1a8b4306dc279d9e6d517/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3c689c4bd9a2f693ad3298bc9c6751fb15e24e768bde0c31bbdc3d60fad12599/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ad236f054fa90d213c3068abbcdd0453ef42bc0c8326a19ea2fb6cc5556a5f38/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/a03eddfe972934d8141a4a5ea5dd63508e92f0d1886727d6553712ce809bc15b/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8a92f668b0c949641709b5fd80d10eaf500cf6d2c34889fdc1dec7ca0d87c9a6/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c57772659693596d049069b226bdae535cfa478de0697b503b97a24c62d88849/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4ccf86bd804b087dc3dad4a14826fec2ef1867a22594e45d730c7fce31c2cc62/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/1deacae372715147166a0f57977f965dcbc8ed00e792124d6953fca7afab33b1/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cbc89cd522b0eea0bbfa31424513893d5f476a65c499729140ab0de8a54b8184/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dac570462938d7cd80fc6f1bd21b505f588b80f980062ecccdffd3566d0310e0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1b4067853d33b4c541ebf455f7733dad1daa5adb3181765bb7c4c98db2b51860/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b2d2b304df1097d0fb74e888396768cf746dcb4e7f12b5ed7938c753874d653d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/03ae1d69356ade2eb960a634a0c84b529fc53da0b203f5495c4db13f89b4426f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ef7cb0a67396e69261ae4a3912456e4bd7f3ef4c27bd6555d779e4089b0799d4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/85183a5e3e526db5b29b8250118d8a3f536ba52a98b95c15f4052cc807183b3f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/9939ccf908ee30f0a58325d9db5aa6c001716acb65c4d30eb5b612d8b47762e7/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a7573d2a25fda8517d99e2024d08178b5321e074bb0574384bc5c3f6ac09153f/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/56f84b29b023476d1d483bb2e9b8c82c00c646582bca8d7f7853e551d6177891/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9727ee03933289707d76f079c56c38c3f004e131ceae97e719e6a83285b0d05d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/68aec81d81efe8470c2a8b24e6903720dc6dedc2b83b3d48eacf4984de6529cf/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a0757f932f70027330aec3cde5804be97beb04c777fde367c1ab3b1514c46769/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/5072642cfda47c975e9ee66ae87251e0bd92dcffaaa9c337f5e9b12537199f3b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/7f20d3aaaed50a6ce2f32d1244e64b730115d2e608447b1acc58c1700fbe47de/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4424cf224f8c80236b22d9c898b875f9761ded13cff6f580d5c88ca1a10134ec/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/93e05f255bcf9410653a8952a180d0f2709af7d0d82000318ab26a7ee695d55d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/4e651bc52b23f54744b27660848f8eb4eb8aa66301c3aa79ef797281c7bddbdf/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/1b9c1ad97e405c307630a633f246bb7b14d2b7804ca19490ceefab85ee554ca8/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/b6daa5a27f4c6b65a83320e340a6b7930b08e939b17bc7d3489cf22f0ef50404/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/12b300bf781c368d4b988cb456060378f5d0a7848bae64f70acc1ce5cccc8154/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/e7c626bbc86b15e5bda3c53252a58f41680b0393602a2b25c99720f3322b4063/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/23f576464c7e1e233eaea5e5c9cb6b6ca69a18d6c7c826bf24afe9a25fc0d68b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/abdd4e486ceb9e23e9ba8624d4ce688d45186b0c992fa7f4f833f81461644265/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/d6cdc0717431f97c2a34eab48622f87577768f5f486ecb7f977fd856bf12ffca/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/b4d2cc28dc10a047415d3b7799a57c5f0363d5a6b4e18d0b30b1c191ce5ccc82/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/def9711b010fb99e734e896a88a237d70545f9b5918d3967d52554d5ddd4da17/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/9832cf33cb66ee1b742ce63a6d72fdc7cf29e7105500a2021d3dd921984d31bf/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/66469df5749965dadb109b1637a898154aac707dcc76f7266824d82039ebaa15/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/21257a838c8c3e7c49c64cd6f1672ae8add04e0de47b688e8b87d8f0a180cd69/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/8b67fbda923aca15aa74824bc4aaf8dee68f11536f357639ae0d05f3e6c23750/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/597aade51e5e60db369a390d2005f2307ae0eb3a648a8e8f883195f2778a07d1/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/7cd7c69532ba422961c9d01ff6ac9b3382d31cad69fe2edffa427bd3527208ce/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/51af0f50451052f3a72bb5f54fbd9ac6fda15108dd267b2d78526f7d64718a3b/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/ff769958163c74b52aa2c327930b9c28bb129157f94e0e298ea2be4209800d82/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/f83350a7f03356ac8cb9751fe47cfcaf27fd76622822c92895d7cd3d5fbe7fbd/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/9f7734d55dbc41297c54943ebec3d293ad6593ec9923b0c81c8c7078600a40f3/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/1ef4b6aef05559c7eda1eb4dcf683a200576839a57c4797f434f4277fefa9bd6/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a3dcb94f1a6127a571747b5b32859312750268fb73938c4b4e976b24718bd332/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4e0933da196a21e5d32af71336f8ff9b834fc6d9d80217a3db672de1c536ae87/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5aff11b3d259ca5e680aeabc384f783cbad20ff0c82b25c35ed44e8b3e2f4ed4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b18e1c809067037bae22adf741e490c0c99ab79e4ece5dea3c14c5fe86218127/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8a34bf33889c7f145adfd744be3c3c54f383e21520bc7a496b451ed1af54d875/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6fbabaaa2ef75514502388773293adb3f1619f3917ce8c525b618f8017241458/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6e906b551cc318ad7e2e14711496c14f02f7ec80bdca7281f4acd1330d89c05f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/7a7082ec9141eb813374aa580728569bd4779be1fdcf7dca9b629c0c8fea39aa/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/722d35785ccdf241783ab3c8f8b20f5dd74b1176bcca4a5e927911e4b4178d3e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 23:28:54.959866 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.959476    5074 manager.go:217] Machine: {Timestamp:2026-11-03 23:28:54.959016525 +0000 UTC m=+0.058616660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100558 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ef07376810e8ec536a66fc98485a SystemUUID:ec21ef07-3768-10e8-ec53-6a66fc98485a BootID:814e9d61-da51-4ad3-bee6-346d5921b62b Filesystems:[{Device:/run/containers/storage/overlay-containers/eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/volumes/kubernetes.io~projected/kube-api-access-scd5g DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a4f4887dd9a47842b5989a926a589244a6e1e573ffff766de9ff50d5bc489b2/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b3e3bc4d2c42ee9576183edcc14026b2ce3708955bd279a8c48c192b0d8cdff/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/442a16eb2ce943c5e4cfa68fd479e0eb188f073c9ea053a005d4bbcc87c2402e/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5032904c-5a18-4f28-be3b-a02375b14113/volumes/kubernetes.io~projected/kube-api-access-wsv66 DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40135659-f74d-4307-8bb3-574411040f5a/volumes/kubernetes.io~projected/kube-api-access-khhwq DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a32ab19d-6150-424c-8ce9-d3c8fcf7f967/volumes/kubernetes.io~projected/kube-api-access-mm94g DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17673e8b-0dec-48d4-8843-65147e22372a/volumes/kubernetes.io~projected/kube-api-access-vw4zq DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/360bef2cc1a9c1a3a62969edfb0abbdd3287387c42757cb44f25722ea6efea1a/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69d478d4-7d64-4a68-823e-29c6b40b4a94/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13b0dc51-f5cc-4afd-821f-c3097be21e80/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2d422d3-fe51-4985-8577-4ddfcc24ba8c/volumes/kubernetes.io~projected/kube-api-access-8662p DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~projected/kube-api-access-425hw DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4741d3bf-14a8-4118-a61e-32dd236755dd/volumes/kubernetes.io~projected/kube-api-access-fkdbc DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb8c88a-1985-4ca8-a11f-7ab86cd87634/volumes/kubernetes.io~projected/kube-api-access-gc8lk DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64bfb43ef6f409286c59d09fe02845562ab21258407667a0d3969afff7548a3e/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d9b50efdd5bbed2f29bf7de9d13d454a607b89fee494995e709ace5a37181ee/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c752b04e-c068-4c8c-94d3-405ff6ab27bb/volumes/kubernetes.io~projected/kube-api-access-9kbjd DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9ae51e161ab6ca1e911d93e0e1aa74b1b9c7cc81eceb5d2963fba14a2dd2c66/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ebc10a97-edbf-42d7-bfd5-caa1d573329e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd9bbc11-d972-4662-bb0d-b1d8744673c9/volumes/kubernetes.io~projected/kube-api-access-98v7q DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/227a6adf06fd3dc6a40a5a0973a5027b77fdcf26d319e08cef23726c2edeb278/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4d8ed01a96cea07661157df7583a36c8a59f66905c164f29ce1b28fd69b7d1b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1e7a5aa-87e3-4afd-ae59-801a559622f6/volumes/kubernetes.io~projected/kube-api-access-gbrrf DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e63f4b31433659d00250c0c21994453bf0c47d787e8afe16b797177a0becbb24/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75649c60-b6dd-4fe0-96f7-ff34c763de46/volumes/kubernetes.io~projected/kube-api-access-hb2gn DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30bd77bc91120979532ef48a2c14cb382635d00331f043d9539f6c7a7ba58435/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/942e7076ee437b48b92e361c9621077801e0f8254e7167f38be99eb1d8afe850/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0dae53-7734-414d-94ca-7bfd0bbc5d62/volumes/kubernetes.io~projected/kube-api-access-smznp DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volumes/kubernetes.io~projected/kube-api-access-bvpvn DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/701dc2a6f25600d147098c94d086e81ffbf795f9b77e9419e9eb412d2a191925/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a23c360-8e0e-4985-ba27-a6631df95e11/volumes/kubernetes.io~projected/kube-api-access-vbhzz DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/volumes/kubernetes.io~projected/kube-api-access-g86bg DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b107666-eaaa-4d08-839b-d3e91dd5868b/volumes/kubernetes.io~projected/kube-api-access-g8vfw DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22bc07b5bc75d1be534a8d9383cec48148964ee9dd0d94256c870fc547e1d064/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1244c649-9fe7-40c5-9b55-443a2529c710/volumes/kubernetes.io~projected/kube-api-access-62llt DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47845d6a-a330-49f4-be05-d3a8aeb677dc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43ed9f0d7ed7bd1acf984dae466e36e7537b9ac4015f5207450cec31d09f2ba9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13b0dc51-f5cc-4afd-821f-c3097be21e80/volumes/kubernetes.io~projected/kube-api-access-c59s2 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb8c88a-1985-4ca8-a11f-7ab86cd87634/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~projected/kube-api-access-9twwh DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0673b7e072e500e715ec6a04abab0c934b8c4d3d3ece1c2fb744ea7d8604b5c/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25859c6e2d2dc8be046891178753e63ee61227467f86caeff5c0fa4bedfa2097/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbbb34c1-9e43-4640-82ae-660e1e365930/volumes/kubernetes.io~projected/kube-api-access-6kmv7 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcd47e4a-ca1b-4557-82d0-200bdb384976/volumes/kubernetes.io~projected/kube-api-access-rcr4z DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/698112a3-0157-43a7-9ea6-30e554d8be51/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59264120-38b0-4c6d-b3be-7d0340db0e34/volumes/kubernetes.io~projected/kube-api-access-bcmx5 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes/kubernetes.io~projected/kube-api-access-m9gr2 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e1e2958ece4a2ae6476c5fbbfe9bc9b303e07e0fe99bd87e6807d239d62f4e6/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4738a53-824f-4a8b-ab86-b160b9f14c0e/volumes/kubernetes.io~projected/kube-api-access-vdjr2 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e691fdd5b6c6ca11d090e6fe1067fec90737668d914a247c3d63b142164de14f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9763223d-ebce-4993-9932-d443787cd21c/volumes/kubernetes.io~projected/kube-api-access-lkwsc DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e18545d8c86047248ba4efb5fe23f534077f7e9266362fd7d81e7757f1a60db/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8d7c518f82d0925be2a712a922f4ccfa02b542ded12d3f3d0857aab597ee067/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/535e658d6525446679dc0378b57769c572372237990354ec7bae0d2476452fa1/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63593602-b03b-47c5-9d74-a95ff2abfddb/volumes/kubernetes.io~projected/kube-api-access-jcl2v DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1f0dd5d-1903-4b2a-830c-05e93d384fda/volumes/kubernetes.io~projected/kube-api-access-k6vm4 DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b249c0323c9bf3f6b7930c479f678fdb9c241f5367dff248d9ade5cec53be111/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69d478d4-7d64-4a68-823e-29c6b40b4a94/volumes/kubernetes.io~projected/kube-api-access-kp6ff DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b107666-eaaa-4d08-839b-d3e91dd5868b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e73b381-beec-43fe-ba3d-460efc605f8b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebc10a97-edbf-42d7-bfd5-caa1d573329e/volumes/kubernetes.io~projected/kube-api-access-ljwhm DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bc8021d-2812-40f5-99ba-85e10f04ef4f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4c5e97a9edd8d8564b422bb5a81c5ebf31ec8d7711f03ca296f8f58c3ddf8cf/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:22bc07b5bc75d1b MacAddress:de:a4:3f:83:f3:a0 Speed:10000 Mtu:8901} {Name:2b3e3bc4d2c42ee MacAddress:16:10:59:b0:88:9d Speed:10000 Mtu:8901} {Name:2e1e2958ece4a2a MacAddress:6e:97:9e:af:b7:48 Speed:10000 Mtu:8901} {Name:360bef2cc1a9c1a MacAddress:c2:3d:f1:20:97:21 Speed:10000 Mtu:8901} {Name:4a4f4887dd9a478 MacAddress:f2:7c:a5:c2:0f:ce Speed:10000 Mtu:8901} {Name:535e658d6525446 MacAddress:ea:41:98:93:69:40 Speed:10000 Mtu:8901} {Name:64bfb43ef6f4092 MacAddress:2a:a7:8c:0a:c4:f3 Speed:10000 Mtu:8901} {Name:6d9b50efdd5bbed MacAddress:56:5c:74:4b:08:6d Speed:10000 Mtu:8901} {Name:701dc2a6f25600d MacAddress:ea:95:54:45:39:0f Speed:10000 Mtu:8901} {Name:942e7076ee437b4 MacAddress:02:70:e0:76:06:07 Speed:10000 Mtu:8901} {Name:a0673b7e072e500 MacAddress:c2:2b:35:37:72:0a Speed:10000 Mtu:8901} {Name:b249c0323c9bf3f MacAddress:a6:07:d6:2f:36:40 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:c2:c2:9d Speed:0 Mtu:9001} {Name:br-int MacAddress:72:69:2f:c3:32:f0 Speed:0 Mtu:8901} {Name:c4c5e97a9edd8d8 MacAddress:ca:23:00:ac:87:df Speed:10000 Mtu:8901} {Name:d8d7c518f82d092 MacAddress:be:e1:d4:ed:ad:48 Speed:10000 Mtu:8901} {Name:e63f4b31433659d MacAddress:ea:72:9e:0e:63:d8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:c2:c2:9d Speed:0 Mtu:9001} {Name:f9ae51e161ab6ca MacAddress:92:bd:37:89:a6:8f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:57:4d:68:2f:70 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:8e:30:ff:a3:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 23:28:54.960050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.959869    5074 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 23:28:54.960050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.959913    5074 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 23:28:54.960210 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960190    5074 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 23:28:54.960348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960211    5074 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-239.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 23:28:54.960391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960351    5074 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 23:28:54.960391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960359    5074 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 23:28:54.960391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960372    5074 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:28:54.960391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960390    5074 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:28:54.960487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960409    5074 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 23:28:54.960572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960563    5074 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 23:28:54.960683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960675    5074 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 23:28:54.960718 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960710    5074 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 23:28:54.960768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960723    5074 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 23:28:54.960768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960732    5074 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 23:28:54.960845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960765    5074 kubelet.go:405] "Adding apiserver pod source"
Nov 03 23:28:54.960845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.960786    5074 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 23:28:54.961494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.961480    5074 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 23:28:54.961494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.961492    5074 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:28:54.961993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.961976    5074 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 23:28:54.964188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964172    5074 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 23:28:54.964188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964187    5074 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:28:54.964331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964310    5074 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 23:28:54.964539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964524    5074 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 23:28:54.964600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964564    5074 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 23:28:54.964690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964679    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 23:28:54.964764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964697    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 23:28:54.964764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964709    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 23:28:54.964764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964720    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 23:28:54.964764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964729    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 23:28:54.964764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964753    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 23:28:54.964764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964762    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964773    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964783    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964794    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964807    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964821    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964856    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 23:28:54.965090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.964865    5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 23:28:54.968777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.968758    5074 server.go:1263] "Started kubelet"
Nov 03 23:28:54.969281 ip-10-0-138-239 systemd[1]: Started Kubernetes Kubelet.
Nov 03 23:28:54.969831 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.969292    5074 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 23:28:54.971149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.971075    5074 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 23:28:54.971254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.971184    5074 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 23:28:54.971545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.971514    5074 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 23:28:54.971730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.971716    5074 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 23:28:54.974971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.974953    5074 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.974975    5074 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.975070    5074 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.975081    5074 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.975091    5074 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:15 +0000 UTC" deadline="2028-08-05 18:13:50.06293381 +0000 UTC"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:54.975108    5074 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-239.us-east-2.compute.internal\" not found"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.975115    5074 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15378h44m55.087822535s"
Nov 03 23:28:54.975319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.975225    5074 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 23:28:54.978785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978757    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="698112a3-0157-43a7-9ea6-30e554d8be51" volumeName="kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-default-certificate" seLinuxMountContext=""
Nov 03 23:28:54.978785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978784    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75649c60-b6dd-4fe0-96f7-ff34c763de46" volumeName="kubernetes.io/projected/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-api-access-hb2gn" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978794    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" volumeName="kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978803    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" volumeName="kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-trusted-ca" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978811    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" volumeName="kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-config" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978819    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b107666-eaaa-4d08-839b-d3e91dd5868b" volumeName="kubernetes.io/secret/3b107666-eaaa-4d08-839b-d3e91dd5868b-cert" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978826    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47845d6a-a330-49f4-be05-d3a8aeb677dc" volumeName="kubernetes.io/secret/47845d6a-a330-49f4-be05-d3a8aeb677dc-agent-certs" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978833    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" volumeName="kubernetes.io/secret/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-serving-cert" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978842    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4741d3bf-14a8-4118-a61e-32dd236755dd" volumeName="kubernetes.io/projected/4741d3bf-14a8-4118-a61e-32dd236755dd-kube-api-access-fkdbc" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978849    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69d478d4-7d64-4a68-823e-29c6b40b4a94" volumeName="kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-service-ca-bundle" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978857    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e7a5aa-87e3-4afd-ae59-801a559622f6" volumeName="kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-daemon-config" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978864    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978879    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978887    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb8c88a-1985-4ca8-a11f-7ab86cd87634" volumeName="kubernetes.io/secret/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-serving-cert" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978896    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69d478d4-7d64-4a68-823e-29c6b40b4a94" volumeName="kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-snapshots" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978904    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978911    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b0dc51-f5cc-4afd-821f-c3097be21e80" volumeName="kubernetes.io/projected/13b0dc51-f5cc-4afd-821f-c3097be21e80-kube-api-access-c59s2" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978918    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0dae53-7734-414d-94ca-7bfd0bbc5d62" volumeName="kubernetes.io/projected/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-kube-api-access-smznp" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978925    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63593602-b03b-47c5-9d74-a95ff2abfddb" volumeName="kubernetes.io/projected/63593602-b03b-47c5-9d74-a95ff2abfddb-kube-api-access-jcl2v" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978940    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="698112a3-0157-43a7-9ea6-30e554d8be51" volumeName="kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-stats-auth" seLinuxMountContext=""
Nov 03 23:28:54.978928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978960    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69d478d4-7d64-4a68-823e-29c6b40b4a94" volumeName="kubernetes.io/projected/69d478d4-7d64-4a68-823e-29c6b40b4a94-kube-api-access-kp6ff" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978971    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978985    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.978996    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0dae53-7734-414d-94ca-7bfd0bbc5d62" volumeName="kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-tmp" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979008    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0dae53-7734-414d-94ca-7bfd0bbc5d62" volumeName="kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-tuned" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979030    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a23c360-8e0e-4985-ba27-a6631df95e11" volumeName="kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979043    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc8021d-2812-40f5-99ba-85e10f04ef4f" volumeName="kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-env-overrides" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979055    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b0dc51-f5cc-4afd-821f-c3097be21e80" volumeName="kubernetes.io/secret/13b0dc51-f5cc-4afd-821f-c3097be21e80-serving-cert" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979071    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb8c88a-1985-4ca8-a11f-7ab86cd87634" volumeName="kubernetes.io/projected/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-kube-api-access-gc8lk" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979084    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47845d6a-a330-49f4-be05-d3a8aeb677dc" volumeName="kubernetes.io/configmap/47845d6a-a330-49f4-be05-d3a8aeb677dc-konnectivity-ca" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979096    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75649c60-b6dd-4fe0-96f7-ff34c763de46" volumeName="kubernetes.io/empty-dir/75649c60-b6dd-4fe0-96f7-ff34c763de46-data-volume" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979108    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d" volumeName="kubernetes.io/configmap/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-nginx-conf" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979122    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979134    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979149    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979160    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979172    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17673e8b-0dec-48d4-8843-65147e22372a" volumeName="kubernetes.io/projected/17673e8b-0dec-48d4-8843-65147e22372a-kube-api-access-vw4zq" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979183    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a23c360-8e0e-4985-ba27-a6631df95e11" volumeName="kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979196    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc8021d-2812-40f5-99ba-85e10f04ef4f" volumeName="kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-config" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979210    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69d478d4-7d64-4a68-823e-29c6b40b4a94" volumeName="kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979222    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69d478d4-7d64-4a68-823e-29c6b40b4a94" volumeName="kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-tmp" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979235    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e7a5aa-87e3-4afd-ae59-801a559622f6" volumeName="kubernetes.io/projected/a1e7a5aa-87e3-4afd-ae59-801a559622f6-kube-api-access-gbrrf" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979248    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b107666-eaaa-4d08-839b-d3e91dd5868b" volumeName="kubernetes.io/projected/3b107666-eaaa-4d08-839b-d3e91dd5868b-kube-api-access-g8vfw" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979262    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e73b381-beec-43fe-ba3d-460efc605f8b" volumeName="kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979274    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" volumeName="kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979281    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" volumeName="kubernetes.io/projected/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-kube-api-access-scd5g" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979289    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcd47e4a-ca1b-4557-82d0-200bdb384976" volumeName="kubernetes.io/projected/fcd47e4a-ca1b-4557-82d0-200bdb384976-kube-api-access-rcr4z" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979300    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b0dc51-f5cc-4afd-821f-c3097be21e80" volumeName="kubernetes.io/configmap/13b0dc51-f5cc-4afd-821f-c3097be21e80-config" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979311    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc8021d-2812-40f5-99ba-85e10f04ef4f" volumeName="kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979324    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="698112a3-0157-43a7-9ea6-30e554d8be51" volumeName="kubernetes.io/projected/698112a3-0157-43a7-9ea6-30e554d8be51-kube-api-access-9twwh" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979337    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32ab19d-6150-424c-8ce9-d3c8fcf7f967" volumeName="kubernetes.io/configmap/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-config-volume" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979349    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40135659-f74d-4307-8bb3-574411040f5a" volumeName="kubernetes.io/configmap/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-config" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979362    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a23c360-8e0e-4985-ba27-a6631df95e11" volumeName="kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979374    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc8021d-2812-40f5-99ba-85e10f04ef4f" volumeName="kubernetes.io/secret/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979386    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9763223d-ebce-4993-9932-d443787cd21c" volumeName="kubernetes.io/projected/9763223d-ebce-4993-9932-d443787cd21c-kube-api-access-lkwsc" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979398    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" volumeName="kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-binary-copy" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979435    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4738a53-824f-4a8b-ab86-b160b9f14c0e" volumeName="kubernetes.io/projected/c4738a53-824f-4a8b-ab86-b160b9f14c0e-kube-api-access-vdjr2" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979449    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcd47e4a-ca1b-4557-82d0-200bdb384976" volumeName="kubernetes.io/configmap/fcd47e4a-ca1b-4557-82d0-200bdb384976-iptables-alerter-script" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979462    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979474    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1244c649-9fe7-40c5-9b55-443a2529c710" volumeName="kubernetes.io/projected/1244c649-9fe7-40c5-9b55-443a2529c710-kube-api-access-62llt" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979486    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e7a5aa-87e3-4afd-ae59-801a559622f6" volumeName="kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cni-binary-copy" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979498    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4738a53-824f-4a8b-ab86-b160b9f14c0e" volumeName="kubernetes.io/empty-dir/c4738a53-824f-4a8b-ab86-b160b9f14c0e-tmp-dir" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979509    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd9bbc11-d972-4662-bb0d-b1d8744673c9" volumeName="kubernetes.io/configmap/cd9bbc11-d972-4662-bb0d-b1d8744673c9-serviceca" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979521    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979533    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc8021d-2812-40f5-99ba-85e10f04ef4f" volumeName="kubernetes.io/projected/5bc8021d-2812-40f5-99ba-85e10f04ef4f-kube-api-access-bvpvn" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979545    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69d478d4-7d64-4a68-823e-29c6b40b4a94" volumeName="kubernetes.io/secret/69d478d4-7d64-4a68-823e-29c6b40b4a94-serving-cert" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979556    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32ab19d-6150-424c-8ce9-d3c8fcf7f967" volumeName="kubernetes.io/projected/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-kube-api-access-mm94g" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979567    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c752b04e-c068-4c8c-94d3-405ff6ab27bb" volumeName="kubernetes.io/projected/c752b04e-c068-4c8c-94d3-405ff6ab27bb-kube-api-access-9kbjd" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979578    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd9bbc11-d972-4662-bb0d-b1d8744673c9" volumeName="kubernetes.io/projected/cd9bbc11-d972-4662-bb0d-b1d8744673c9-kube-api-access-98v7q" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979589    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc10a97-edbf-42d7-bfd5-caa1d573329e" volumeName="kubernetes.io/secret/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-key" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979601    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979612    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb8c88a-1985-4ca8-a11f-7ab86cd87634" volumeName="kubernetes.io/configmap/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-config" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979623    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5032904c-5a18-4f28-be3b-a02375b14113" volumeName="kubernetes.io/projected/5032904c-5a18-4f28-be3b-a02375b14113-kube-api-access-wsv66" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979634    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2d422d3-fe51-4985-8577-4ddfcc24ba8c" volumeName="kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979645    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40135659-f74d-4307-8bb3-574411040f5a" volumeName="kubernetes.io/projected/40135659-f74d-4307-8bb3-574411040f5a-kube-api-access-khhwq" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979657    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a23c360-8e0e-4985-ba27-a6631df95e11" volumeName="kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979676    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" volumeName="kubernetes.io/projected/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-kube-api-access-g86bg" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979688    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc10a97-edbf-42d7-bfd5-caa1d573329e" volumeName="kubernetes.io/configmap/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-cabundle" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979700    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc10a97-edbf-42d7-bfd5-caa1d573329e" volumeName="kubernetes.io/projected/ebc10a97-edbf-42d7-bfd5-caa1d573329e-kube-api-access-ljwhm" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979711    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01084ce5-4229-4c46-bb4e-9c6973a000e4" volumeName="kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979723    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4741d3bf-14a8-4118-a61e-32dd236755dd" volumeName="kubernetes.io/configmap/4741d3bf-14a8-4118-a61e-32dd236755dd-telemetry-config" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979734    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1f0dd5d-1903-4b2a-830c-05e93d384fda" volumeName="kubernetes.io/projected/a1f0dd5d-1903-4b2a-830c-05e93d384fda-kube-api-access-k6vm4" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979766    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32ab19d-6150-424c-8ce9-d3c8fcf7f967" volumeName="kubernetes.io/empty-dir/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-tmp-dir" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979778    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbb34c1-9e43-4640-82ae-660e1e365930" volumeName="kubernetes.io/projected/bbbb34c1-9e43-4640-82ae-660e1e365930-kube-api-access-6kmv7" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979790    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" volumeName="kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979801    5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59264120-38b0-4c6d-b3be-7d0340db0e34" volumeName="kubernetes.io/projected/59264120-38b0-4c6d-b3be-7d0340db0e34-kube-api-access-bcmx5" seLinuxMountContext=""
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979813    5074 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.979820    5074 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980109    5074 factory.go:153] Registering CRI-O factory
Nov 03 23:28:54.980081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980126    5074 factory.go:223] Registration of the crio container factory successfully
Nov 03 23:28:54.983581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980196    5074 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 23:28:54.983581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980207    5074 factory.go:55] Registering systemd factory
Nov 03 23:28:54.983581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980216    5074 factory.go:223] Registration of the systemd container factory successfully
Nov 03 23:28:54.983581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980235    5074 factory.go:103] Registering Raw factory
Nov 03 23:28:54.983581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980251    5074 manager.go:1199] Started watching for new ooms in manager
Nov 03 23:28:54.983581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.980551    5074 manager.go:319] Starting recovery of all containers
Nov 03 23:28:54.985288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.985262    5074 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:54.987128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.986892    5074 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:54.988975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.988312    5074 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:54.988975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.988373    5074 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 23:28:54.997463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.997441    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:28:54.999555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:54.999534    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:28:55.006333 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.006013    5074 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 23:28:55.009147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.009125    5074 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 23:28:55.009147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.009150    5074 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 23:28:55.009289 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.009172    5074 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 23:28:55.009289 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:55.009216    5074 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 23:28:55.014600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.014581    5074 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:55.026008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.025971    5074 generic.go:356] "Generic (PLEG): container finished" podID="a9a45f1006b1620e684bf8ebed402834" containerID="b5ea30da3679c0290a89ecd50f10f220485a6d2038440c0198745f6b7b3b080f" exitCode=0
Nov 03 23:28:55.051887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.051824    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:28:55.051887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.051860    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb" containerID="3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495" exitCode=255
Nov 03 23:28:55.055757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.055714    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="9a08842926263ee132242508a6c9c30af474d539f787e9ad58f665949ca6a6c6" exitCode=0
Nov 03 23:28:55.055757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.055758    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="e16ea40b2ec00f27e88099f72e0f8c55790dfe7e47c62612a2fc19767767d75f" exitCode=0
Nov 03 23:28:55.055919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.055767    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="b12e05776bd12a9bef0293ea01409cbbf7a83f03d1d60768b3ca1515e45e4ae4" exitCode=0
Nov 03 23:28:55.055919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.055773    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="1d526e5703b562cef09912248098bb924efbb325ecc77d6eb8951f517a227db6" exitCode=0
Nov 03 23:28:55.055919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.055778    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="440884e102042d8b584a393b636d7bcf42b53e53d46237fd7df2608bb9468bbf" exitCode=0
Nov 03 23:28:55.055919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.055783    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bbd0ed-0bf6-401c-8ea6-d759c3909adc" containerID="e8009b352aa2a547ecb311d91f279f083bdbccb06b496a4256bcdf7bca98bfe1" exitCode=0
Nov 03 23:28:55.087140 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.087115    5074 manager.go:324] Recovery completed
Nov 03 23:28:55.108848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.108823    5074 cpu_manager.go:226] "Starting" policy="none"
Nov 03 23:28:55.108848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.108842    5074 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 23:28:55.109015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.108862    5074 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 23:28:55.109015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.108980    5074 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 23:28:55.109429 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.108991    5074 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 23:28:55.109472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109433    5074 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 23:28:55.109472 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:55.109432    5074 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 23:28:55.109472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109443    5074 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 23:28:55.109472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109451    5074 policy_none.go:50] "Start"
Nov 03 23:28:55.109472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109462    5074 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 23:28:55.109606 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109476    5074 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 23:28:55.109635 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109607    5074 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 23:28:55.109635 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109614    5074 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 23:28:55.109635 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.109619    5074 policy_none.go:44] "Start"
Nov 03 23:28:55.114075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.114050    5074 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 23:28:55.116610 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.116590    5074 manager.go:341] "Starting Device Plugin manager"
Nov 03 23:28:55.116717 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:55.116620    5074 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 23:28:55.116717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.116631    5074 server.go:87] "Starting device plugin registration server"
Nov 03 23:28:55.116962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.116949    5074 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 23:28:55.117013 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.116964    5074 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 23:28:55.117132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.117106    5074 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 23:28:55.117258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.117217    5074 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 23:28:55.117258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.117258    5074 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 23:28:55.117363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.117264    5074 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 23:28:55.117413 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.117308    5074 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T23:28:55.117246224Z","UUID":"1738c10c-0ac4-400f-a0c8-378896a0a417","Handler":null,"Name":"","Endpoint":""}
Nov 03 23:28:55.118479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.118463    5074 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 23:28:55.118558 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.118490    5074 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 23:28:55.118800 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:55.118763    5074 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 23:28:55.217365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.217328    5074 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:55.218802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.218783    5074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:55.218861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.218827    5074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:55.218861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.218837    5074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-239.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:55.218916 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.218881    5074 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.241360 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.241335    5074 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.241465 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.241405    5074 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.310191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310103    5074 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"]
Nov 03 23:28:55.310684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310654    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"a9a45f1006b1620e684bf8ebed402834","Type":"ContainerStarted","Data":"0350f1acdb7dfec8473f5e05475f587c299316f40567460b5bcb10ebe5de03f1"}
Nov 03 23:28:55.310724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310692    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"a9a45f1006b1620e684bf8ebed402834","Type":"ContainerDied","Data":"b5ea30da3679c0290a89ecd50f10f220485a6d2038440c0198745f6b7b3b080f"}
Nov 03 23:28:55.310724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310708    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"a9a45f1006b1620e684bf8ebed402834","Type":"ContainerStarted","Data":"eb9cbb8c9c967e76aa52f72144e39ad3f0b07970209f7f168c8a09fdb3196c63"}
Nov 03 23:28:55.310869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310804    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"14f459d1824c1ea630aa90a99306c647","Type":"ContainerStarted","Data":"08d36c0423e39b8aed45905bfc02f8eaddedadc910a336c904191d342816ef3d"}
Nov 03 23:28:55.310869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310818    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" event={"ID":"14f459d1824c1ea630aa90a99306c647","Type":"ContainerStarted","Data":"78a95cdf8fa790817ff72f6465b95b6f4cf8b08d6d22a00975d8da0004661170"}
Nov 03 23:28:55.310869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310820    5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.310998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.310885    5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.325954 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.325934    5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:55.326023 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:55.325976    5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.326603 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.326588    5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:55.326672 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:55.326628    5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.392024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.391984    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.392024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.392025    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.392220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.392051    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14f459d1824c1ea630aa90a99306c647-config\") pod \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"14f459d1824c1ea630aa90a99306c647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.492382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.492346    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.492382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.492389    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.492578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.492410    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14f459d1824c1ea630aa90a99306c647-config\") pod \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"14f459d1824c1ea630aa90a99306c647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.492578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.492451    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14f459d1824c1ea630aa90a99306c647-config\") pod \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"14f459d1824c1ea630aa90a99306c647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.492578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.492459    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.492578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.492459    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9a45f1006b1620e684bf8ebed402834-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" (UID: \"a9a45f1006b1620e684bf8ebed402834\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:55.961530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.961483    5074 apiserver.go:52] "Watching apiserver"
Nov 03 23:28:55.983138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.983094    5074 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 23:28:55.983979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.983940    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl","openshift-insights/insights-runtime-extractor-dzv94","openshift-service-ca/service-ca-5fffc97bdf-d5bgc","kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-qjdfr","kube-system/kas-connection-checker-685895c7c9-c6qhd","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj","openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48","kube-system/kas-connection-checker-685895c7c9-6mxbn","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h","openshift-image-registry/node-ca-cdxlq","openshift-dns/node-resolver-c99qk","openshift-insights/periodic-gathering-ddq79-rh2q9","openshift-network-diagnostics/network-check-target-7cjjg","openshift-network-console/networking-console-plugin-86c478bb88-cs8jk","openshift-ingress/router-default-5bb58b79c9-bksx8","openshift-ovn-kubernetes/ovnkube-node-ncc9n","openshift-insights/insights-operator-7c4986d449-7869h","openshift-image-registry/image-registry-5b96cb8d6-ctr95","openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg","openshift-ingress-canary/ingress-canary-6mthf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4","openshift-multus/multus-jt2sr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-pcvlc","openshift-cluster-node-tuning-operator/tuned-dxj77","openshift-image-registry/image-registry-6768769b75-2ms22","openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v","openshift-multus/network-metrics-daemon-6n2fd","openshift-network-operator/iptables-alerter-6czcz","openshift-multus/multus-additional-cni-plugins-cqzg8","openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc","openshift-dns/dns-default-4x94t","kube-system/konnectivity-agent-b9jnr","openshift-console-operator/console-operator-7cd56b8dbc-6sdkj","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"]
Nov 03 23:28:55.985279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.985261    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:55.987173 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.987146    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:55.987471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.987456    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:55.987814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.987792    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:55.989148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.989126    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:55.991651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.991631    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:55.999319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.999287    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:55.999843 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:55.999820    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.001546 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.001519    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.002249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.002213    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 23:28:56.002249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.002215    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nk9gg\""
Nov 03 23:28:56.002441 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.002252    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.003463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.003440    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 23:28:56.003613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.003491    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.003613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.003578    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.003732 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.003716    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.003883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.003825    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.004087 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004066    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-78rrm\""
Nov 03 23:28:56.004213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004195    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 23:28:56.004525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004501    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 23:28:56.004628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004538    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.004628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004553    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.004628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004566    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.004628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004572    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.004628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.004539    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.006125 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.006107    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.006230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.006133    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.006230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.006185    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 23:28:56.007057 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.007038    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 23:28:56.007121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.007075    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.007121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.007095    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.009354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.009338    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.010720 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.010703    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.010795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.010703    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.018896 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.018870    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 23:28:56.019544 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.019507    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 23:28:56.019646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.019556    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 23:28:56.021982 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.021959    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 23:28:56.022072 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.021964    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 23:28:56.023918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.023904    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q2zjl\""
Nov 03 23:28:56.023983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.023927    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xj6sl\""
Nov 03 23:28:56.024505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.024474    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ws2sq\""
Nov 03 23:28:56.024505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.024506    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 23:28:56.024729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.024578    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 23:28:56.024729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.024581    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 23:28:56.024729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.024589    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9j4q5\""
Nov 03 23:28:56.024729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.024599    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t7fl4\""
Nov 03 23:28:56.031889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.031858    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 23:28:56.032334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.032316    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 23:28:56.032407 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.032368    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 23:28:56.066979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.066914    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.067160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.066990    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.067468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067439    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067478    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067489    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067551    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067566    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067570    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067609    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 23:28:56.067614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067616    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067611    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067633    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067643    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067661    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067667    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067682    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067705    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067713    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067725    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067634    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067768    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067794    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067834    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067883    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067888    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067908    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067922    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067928    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.068009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.067970    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.069427 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069403    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 23:28:56.069427 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069408    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.069427 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069428    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.069657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069409    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.069657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069448    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:56.069657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069456    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 23:28:56.069657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069468    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 23:28:56.069657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069469    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 23:28:56.069657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.069502    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 23:28:56.078579 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.078546    5074 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 23:28:56.080343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.080321    5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:56.080527 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.080512    5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:56.093368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.093336    5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:56.093575 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.093474    5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:56.093783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.093766    5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:56.093877 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.093822    5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-239.us-east-2.compute.internal"
Nov 03 23:28:56.095007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.094979    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4738a53-824f-4a8b-ab86-b160b9f14c0e-hosts-file\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.095152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095013    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-config\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:56.095152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095031    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98v7q\" (UniqueName: \"kubernetes.io/projected/cd9bbc11-d972-4662-bb0d-b1d8744673c9-kube-api-access-98v7q\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.095152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095048    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-systemd\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.095152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095112    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75649c60-b6dd-4fe0-96f7-ff34c763de46-data-volume\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.095319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095171    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cnibin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.095319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095199    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75649c60-b6dd-4fe0-96f7-ff34c763de46-data-volume\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.095319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095247    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-conf-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.095319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095305    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-modprobe-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.095491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095314    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-config\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:56.095491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095374    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysconfig\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.095491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095402    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 23:28:56.095491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095413    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twwh\" (UniqueName: \"kubernetes.io/projected/698112a3-0157-43a7-9ea6-30e554d8be51-kube-api-access-9twwh\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.095491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095459    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-etc-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.095686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095512    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr4z\" (UniqueName: \"kubernetes.io/projected/fcd47e4a-ca1b-4557-82d0-200bdb384976-kube-api-access-rcr4z\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.095686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095531    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.095686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095554    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-systemd-units\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.095686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095604    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 23:28:56.095862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095771    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-sys-fs\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.095862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095806    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpvn\" (UniqueName: \"kubernetes.io/projected/5bc8021d-2812-40f5-99ba-85e10f04ef4f-kube-api-access-bvpvn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.095862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095833    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-serving-cert\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.096060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095859    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cnibin\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.096060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095887    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47845d6a-a330-49f4-be05-d3a8aeb677dc-agent-certs\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:56.096060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095915    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.096060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095941    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-tuned\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.096060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.095972    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75649c60-b6dd-4fe0-96f7-ff34c763de46-crio-socket\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.096275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096063    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-tuned\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.096275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096065    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.096275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096093    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-conf\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.096275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096177    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-lib-modules\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.096275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096205    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-stats-auth\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096232    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-etc-kubernetes\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096315    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbjd\" (UniqueName: \"kubernetes.io/projected/c752b04e-c068-4c8c-94d3-405ff6ab27bb-kube-api-access-9kbjd\") pod \"migrator-66df5d496b-nls48\" (UID: \"c752b04e-c068-4c8c-94d3-405ff6ab27bb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096330    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-serving-cert\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096360    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm94g\" (UniqueName: \"kubernetes.io/projected/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-kube-api-access-mm94g\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096372    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096439    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 23:28:56.096472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096449    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096481    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\""
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096496    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-slash\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096561    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwn6\""
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096577    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-netd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096636    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljwhm\" (UniqueName: \"kubernetes.io/projected/ebc10a97-edbf-42d7-bfd5-caa1d573329e-kube-api-access-ljwhm\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096664    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-run\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096681    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.096724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096718    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59s2\" (UniqueName: \"kubernetes.io/projected/13b0dc51-f5cc-4afd-821f-c3097be21e80-kube-api-access-c59s2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:56.096957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096837    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:56.096957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096861    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-systemd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.096957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.096918    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-script-lib\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.097040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097014    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.097103 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097082    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.097178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097158    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6ff\" (UniqueName: \"kubernetes.io/projected/69d478d4-7d64-4a68-823e-29c6b40b4a94-kube-api-access-kp6ff\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.097223 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097197    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4738a53-824f-4a8b-ab86-b160b9f14c0e-tmp-dir\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.097272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097220    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-script-lib\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.097353 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097334    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4738a53-824f-4a8b-ab86-b160b9f14c0e-tmp-dir\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.097353 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097344    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.097450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097387    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.097450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097394    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.097450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097420    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-kubelet\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.097450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097413    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097475    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-system-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097555    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-k8s-cni-cncf-io\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097585    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097587    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-hostroot\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097625    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z6886\""
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097646    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-daemon-config\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.097683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097673    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-multus-certs\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097701    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwsc\" (UniqueName: \"kubernetes.io/projected/9763223d-ebce-4993-9932-d443787cd21c-kube-api-access-lkwsc\") pod \"migrator-66df5d496b-bs5gg\" (UID: \"9763223d-ebce-4993-9932-d443787cd21c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097726    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-config-volume\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097838    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scd5g\" (UniqueName: \"kubernetes.io/projected/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-kube-api-access-scd5g\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097876    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097904    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097964    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-config-volume\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.098023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.097969    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-kubernetes\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098027    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-daemon-config\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098031    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vm4\" (UniqueName: \"kubernetes.io/projected/a1f0dd5d-1903-4b2a-830c-05e93d384fda-kube-api-access-k6vm4\") pod \"kas-connection-checker-685895c7c9-qjdfr\" (UID: \"a1f0dd5d-1903-4b2a-830c-05e93d384fda\") " pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098142    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098176    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098227    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098238    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-ovn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098250    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgkz4\""
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098266    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-node-log\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098288    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.098308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098306    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-key\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098341    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-multus\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098420    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-cabundle\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098453    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098506    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098508    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098516    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-key\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098535    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098548    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kpv5j\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098455    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-serving-cert\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098568    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dd24d\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098576    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098572    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098591    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rg97z\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098596    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k24zf\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098629    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098593    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098667    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmv7\" (UniqueName: \"kubernetes.io/projected/bbbb34c1-9e43-4640-82ae-660e1e365930-kube-api-access-6kmv7\") pod \"kas-connection-checker-685895c7c9-6mxbn\" (UID: \"bbbb34c1-9e43-4640-82ae-660e1e365930\") " pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098697    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9hzk\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098701    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ebc10a97-edbf-42d7-bfd5-caa1d573329e-signing-cabundle\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098695    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhwq\" (UniqueName: \"kubernetes.io/projected/40135659-f74d-4307-8bb3-574411040f5a-kube-api-access-khhwq\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098764    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zb5ww\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098712    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098767    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098782    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m6pk6\""
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098770    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-netns\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.098779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098798    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098829    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k4d88\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098837    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q987z\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098854    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jctss\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098877    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w5s9r\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098832    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098915    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-swp8v\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098928    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098942    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b74w2\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098949    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lsnsx\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.098973    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099015    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcd47e4a-ca1b-4557-82d0-200bdb384976-iptables-alerter-script\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099016    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099031    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2bch\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099112    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099148    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cni-binary-copy\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099171    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b107666-eaaa-4d08-839b-d3e91dd5868b-cert\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099196    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099217    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovn-node-metrics-cert\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099235    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d478d4-7d64-4a68-823e-29c6b40b4a94-serving-cert\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099260    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099299    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099320    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcd47e4a-ca1b-4557-82d0-200bdb384976-iptables-alerter-script\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099343    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099357    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cni-binary-copy\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099432    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099495    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b107666-eaaa-4d08-839b-d3e91dd5868b-cert\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099530    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-log-socket\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099549    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099555    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d478d4-7d64-4a68-823e-29c6b40b4a94-serving-cert\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099552    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099571    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099589    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd9bbc11-d972-4662-bb0d-b1d8744673c9-serviceca\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099610    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099617    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-host\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099660    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-device-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099681    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrrf\" (UniqueName: \"kubernetes.io/projected/a1e7a5aa-87e3-4afd-ae59-801a559622f6-kube-api-access-gbrrf\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099699    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099700    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovn-node-metrics-cert\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.099830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099866    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.099913    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd9bbc11-d972-4662-bb0d-b1d8744673c9-serviceca\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100152    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd47e4a-ca1b-4557-82d0-200bdb384976-host-slash\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100198    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-config\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100219    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-snapshots\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100235    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-service-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100254    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100287    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd9bbc11-d972-4662-bb0d-b1d8744673c9-host\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100312    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjr2\" (UniqueName: \"kubernetes.io/projected/c4738a53-824f-4a8b-ab86-b160b9f14c0e-kube-api-access-vdjr2\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100329    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-snapshots\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100334    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-tmp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100375    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-config\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100389    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-kubelet\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100438    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-tmp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100472    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100501    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmx5\" (UniqueName: \"kubernetes.io/projected/59264120-38b0-4c6d-b3be-7d0340db0e34-kube-api-access-bcmx5\") pod \"volume-data-source-validator-58c9f5f46d-svxq9\" (UID: \"59264120-38b0-4c6d-b3be-7d0340db0e34\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100546    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vfw\" (UniqueName: \"kubernetes.io/projected/3b107666-eaaa-4d08-839b-d3e91dd5868b-kube-api-access-g8vfw\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100576    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13b0dc51-f5cc-4afd-821f-c3097be21e80-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100597    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100610    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100625    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-dbus\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100651    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-var-lib-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100678    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-config\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100707    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smznp\" (UniqueName: \"kubernetes.io/projected/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-kube-api-access-smznp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100752    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdbc\" (UniqueName: \"kubernetes.io/projected/4741d3bf-14a8-4118-a61e-32dd236755dd-kube-api-access-fkdbc\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100810    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100853    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100888    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d478d4-7d64-4a68-823e-29c6b40b4a94-service-ca-bundle\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100919    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13b0dc51-f5cc-4afd-821f-c3097be21e80-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100951    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.100976    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101010    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8lk\" (UniqueName: \"kubernetes.io/projected/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-kube-api-access-gc8lk\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101026    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-ovnkube-config\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101127    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101139    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101194    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcl2v\" (UniqueName: \"kubernetes.io/projected/63593602-b03b-47c5-9d74-a95ff2abfddb-kube-api-access-jcl2v\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101212    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-socket-dir-parent\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101216    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101233    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-sys\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101333    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101325    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101354    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-bin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101395    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-default-certificate\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.101456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101491    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101531    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-kubelet-config\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101560    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62llt\" (UniqueName: \"kubernetes.io/projected/1244c649-9fe7-40c5-9b55-443a2529c710-kube-api-access-62llt\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101578    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-netns\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101618    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-registration-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101644    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101644    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-default-certificate\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101672    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101704    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-tmp-dir\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101754    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101816    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-tmp-dir\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101849    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101887    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsv66\" (UniqueName: \"kubernetes.io/projected/5032904c-5a18-4f28-be3b-a02375b14113-kube-api-access-wsv66\") pod \"kas-connection-checker-685895c7c9-c6qhd\" (UID: \"5032904c-5a18-4f28-be3b-a02375b14113\") " pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101905    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47845d6a-a330-49f4-be05-d3a8aeb677dc-konnectivity-ca\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.101954    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13b0dc51-f5cc-4afd-821f-c3097be21e80-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102006    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102050    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102078    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-var-lib-kubelet\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102092    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47845d6a-a330-49f4-be05-d3a8aeb677dc-konnectivity-ca\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102156    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102182    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102218    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4741d3bf-14a8-4118-a61e-32dd236755dd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102236    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102236    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102254    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102276    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-env-overrides\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102294    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g86bg\" (UniqueName: \"kubernetes.io/projected/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-kube-api-access-g86bg\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102310    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-tmp\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102332    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-socket-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e73b381-beec-43fe-ba3d-460efc605f8b-original-pull-secret\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102373    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13b0dc51-f5cc-4afd-821f-c3097be21e80-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102418    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d478d4-7d64-4a68-823e-29c6b40b4a94-tmp\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102453    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4741d3bf-14a8-4118-a61e-32dd236755dd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102475    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc8021d-2812-40f5-99ba-85e10f04ef4f-env-overrides\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102477    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2gn\" (UniqueName: \"kubernetes.io/projected/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-api-access-hb2gn\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102519    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102542    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4zq\" (UniqueName: \"kubernetes.io/projected/17673e8b-0dec-48d4-8843-65147e22372a-kube-api-access-vw4zq\") pod \"network-check-source-6f78d59f99-xr79v\" (UID: \"17673e8b-0dec-48d4-8843-65147e22372a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102576    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-bin\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102593    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102598    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102623    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102646    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102687    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.103153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.102730    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.106349    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47845d6a-a330-49f4-be05-d3a8aeb677dc-agent-certs\") pod \"konnectivity-agent-b9jnr\" (UID: \"47845d6a-a330-49f4-be05-d3a8aeb677dc\") " pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.106502    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-stats-auth\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.108537    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.108807    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-serving-cert\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.111607    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98v7q\" (UniqueName: \"kubernetes.io/projected/cd9bbc11-d972-4662-bb0d-b1d8744673c9-kube-api-access-98v7q\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.112073    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twwh\" (UniqueName: \"kubernetes.io/projected/698112a3-0157-43a7-9ea6-30e554d8be51-kube-api-access-9twwh\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.116969    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr4z\" (UniqueName: \"kubernetes.io/projected/fcd47e4a-ca1b-4557-82d0-200bdb384976-kube-api-access-rcr4z\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.117618    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.118347    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpvn\" (UniqueName: \"kubernetes.io/projected/5bc8021d-2812-40f5-99ba-85e10f04ef4f-kube-api-access-bvpvn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.118946    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g86bg\" (UniqueName: \"kubernetes.io/projected/e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb-kube-api-access-g86bg\") pod \"console-operator-7cd56b8dbc-6sdkj\" (UID: \"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.120150    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmx5\" (UniqueName: \"kubernetes.io/projected/59264120-38b0-4c6d-b3be-7d0340db0e34-kube-api-access-bcmx5\") pod \"volume-data-source-validator-58c9f5f46d-svxq9\" (UID: \"59264120-38b0-4c6d-b3be-7d0340db0e34\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-svxq9"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.120214    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrrf\" (UniqueName: \"kubernetes.io/projected/a1e7a5aa-87e3-4afd-ae59-801a559622f6-kube-api-access-gbrrf\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.122146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.120825    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljwhm\" (UniqueName: \"kubernetes.io/projected/ebc10a97-edbf-42d7-bfd5-caa1d573329e-kube-api-access-ljwhm\") pod \"service-ca-5fffc97bdf-d5bgc\" (UID: \"ebc10a97-edbf-42d7-bfd5-caa1d573329e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d5bgc"
Nov 03 23:28:56.123937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.123712    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbjd\" (UniqueName: \"kubernetes.io/projected/c752b04e-c068-4c8c-94d3-405ff6ab27bb-kube-api-access-9kbjd\") pod \"migrator-66df5d496b-nls48\" (UID: \"c752b04e-c068-4c8c-94d3-405ff6ab27bb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nls48"
Nov 03 23:28:56.123937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.123726    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm94g\" (UniqueName: \"kubernetes.io/projected/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-kube-api-access-mm94g\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.123937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.123877    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62llt\" (UniqueName: \"kubernetes.io/projected/1244c649-9fe7-40c5-9b55-443a2529c710-kube-api-access-62llt\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:56.123937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.123897    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwsc\" (UniqueName: \"kubernetes.io/projected/9763223d-ebce-4993-9932-d443787cd21c-kube-api-access-lkwsc\") pod \"migrator-66df5d496b-bs5gg\" (UID: \"9763223d-ebce-4993-9932-d443787cd21c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bs5gg"
Nov 03 23:28:56.124264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.123971    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.124264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.124173    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6ff\" (UniqueName: \"kubernetes.io/projected/69d478d4-7d64-4a68-823e-29c6b40b4a94-kube-api-access-kp6ff\") pod \"insights-operator-7c4986d449-7869h\" (UID: \"69d478d4-7d64-4a68-823e-29c6b40b4a94\") " pod="openshift-insights/insights-operator-7c4986d449-7869h"
Nov 03 23:28:56.124465 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.124446    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vfw\" (UniqueName: \"kubernetes.io/projected/3b107666-eaaa-4d08-839b-d3e91dd5868b-kube-api-access-g8vfw\") pod \"ingress-canary-6mthf\" (UID: \"3b107666-eaaa-4d08-839b-d3e91dd5868b\") " pod="openshift-ingress-canary/ingress-canary-6mthf"
Nov 03 23:28:56.124693 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.124674    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdbc\" (UniqueName: \"kubernetes.io/projected/4741d3bf-14a8-4118-a61e-32dd236755dd-kube-api-access-fkdbc\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.125178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.125160    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vm4\" (UniqueName: \"kubernetes.io/projected/a1f0dd5d-1903-4b2a-830c-05e93d384fda-kube-api-access-k6vm4\") pod \"kas-connection-checker-685895c7c9-qjdfr\" (UID: \"a1f0dd5d-1903-4b2a-830c-05e93d384fda\") " pod="kube-system/kas-connection-checker-685895c7c9-qjdfr"
Nov 03 23:28:56.126887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.126857    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smznp\" (UniqueName: \"kubernetes.io/projected/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-kube-api-access-smznp\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.126988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.126899    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59s2\" (UniqueName: \"kubernetes.io/projected/13b0dc51-f5cc-4afd-821f-c3097be21e80-kube-api-access-c59s2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj\" (UID: \"13b0dc51-f5cc-4afd-821f-c3097be21e80\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj"
Nov 03 23:28:56.126988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.126930    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") pod \"periodic-gathering-ddq79-rh2q9\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") " pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:28:56.128531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.128508    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8lk\" (UniqueName: \"kubernetes.io/projected/3cb8c88a-1985-4ca8-a11f-7ab86cd87634-kube-api-access-gc8lk\") pod \"service-ca-operator-544d749c8c-8smkc\" (UID: \"3cb8c88a-1985-4ca8-a11f-7ab86cd87634\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc"
Nov 03 23:28:56.128906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.128886    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmv7\" (UniqueName: \"kubernetes.io/projected/bbbb34c1-9e43-4640-82ae-660e1e365930-kube-api-access-6kmv7\") pod \"kas-connection-checker-685895c7c9-6mxbn\" (UID: \"bbbb34c1-9e43-4640-82ae-660e1e365930\") " pod="kube-system/kas-connection-checker-685895c7c9-6mxbn"
Nov 03 23:28:56.129243 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.129221    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcl2v\" (UniqueName: \"kubernetes.io/projected/63593602-b03b-47c5-9d74-a95ff2abfddb-kube-api-access-jcl2v\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.129319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.129221    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.129434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.129415    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhwq\" (UniqueName: \"kubernetes.io/projected/40135659-f74d-4307-8bb3-574411040f5a-kube-api-access-khhwq\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.129930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.129904    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4zq\" (UniqueName: \"kubernetes.io/projected/17673e8b-0dec-48d4-8843-65147e22372a-kube-api-access-vw4zq\") pod \"network-check-source-6f78d59f99-xr79v\" (UID: \"17673e8b-0dec-48d4-8843-65147e22372a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xr79v"
Nov 03 23:28:56.130668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.130647    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scd5g\" (UniqueName: \"kubernetes.io/projected/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-kube-api-access-scd5g\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.135131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.130781    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjr2\" (UniqueName: \"kubernetes.io/projected/c4738a53-824f-4a8b-ab86-b160b9f14c0e-kube-api-access-vdjr2\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.135131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.131128    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8662p\" (UniqueName: \"kubernetes.io/projected/d2d422d3-fe51-4985-8577-4ddfcc24ba8c-kube-api-access-8662p\") pod \"network-check-target-7cjjg\" (UID: \"d2d422d3-fe51-4985-8577-4ddfcc24ba8c\") " pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:56.136086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.136055    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.136188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.136110    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2gn\" (UniqueName: \"kubernetes.io/projected/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-api-access-hb2gn\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.136256 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.136237    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsv66\" (UniqueName: \"kubernetes.io/projected/5032904c-5a18-4f28-be3b-a02375b14113-kube-api-access-wsv66\") pod \"kas-connection-checker-685895c7c9-c6qhd\" (UID: \"5032904c-5a18-4f28-be3b-a02375b14113\") " pod="kube-system/kas-connection-checker-685895c7c9-c6qhd"
Nov 03 23:28:56.203580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203537    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-lib-modules\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.203580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203579    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-etc-kubernetes\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203598    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203617    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-slash\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203635    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-netd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203657    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-run\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203679    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203679    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-etc-kubernetes\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203700    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-systemd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203734    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-netd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203773    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-systemd\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203787    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203775    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203819    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-run\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203827    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-kubelet\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203734    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-slash\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203852    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-system-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203866    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-kubelet\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.203869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203875    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-k8s-cni-cncf-io\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203893    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203895    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-hostroot\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203914    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-system-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203956    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-multus-certs\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203982    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-kubernetes\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204019    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204025    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-k8s-cni-cncf-io\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204043    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-ovn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.203937    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-lib-modules\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204067    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-hostroot\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204109    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-node-log\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204137    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-multus\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204169    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-kubernetes\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204192    5074 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204280    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-node-log\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204276    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204296    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-multus\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204351    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-ovn\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204381    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-multus-certs\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204383    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-netns\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204413    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-run-netns\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204421    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204463    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204493    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.204507    5074 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204525    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-log-socket\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204538    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204558    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.204591    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:56.70456745 +0000 UTC m=+1.804167599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : secret "metrics-daemon-secret" not found
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204624    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-log-socket\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204626    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-host\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.204636 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.204666    5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204693    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-device-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.204733    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:56.704716008 +0000 UTC m=+1.804316169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204773    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-device-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204795    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd47e4a-ca1b-4557-82d0-200bdb384976-host-slash\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204830    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd9bbc11-d972-4662-bb0d-b1d8744673c9-host\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204850    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-host\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204858    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-kubelet\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.204953    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-kubelet\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205002    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-dbus\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205031    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-var-lib-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205067    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205094    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-socket-dir-parent\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-var-lib-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205120    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-sys\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205148    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205095    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-dbus\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205157    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-socket-dir-parent\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205188    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-cni-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205221    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-sys\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205232    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205246    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd47e4a-ca1b-4557-82d0-200bdb384976-host-slash\") pod \"iptables-alerter-6czcz\" (UID: \"fcd47e4a-ca1b-4557-82d0-200bdb384976\") " pod="openshift-network-operator/iptables-alerter-6czcz"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205291    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-bin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205303    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd9bbc11-d972-4662-bb0d-b1d8744673c9-host\") pod \"node-ca-cdxlq\" (UID: \"cd9bbc11-d972-4662-bb0d-b1d8744673c9\") " pod="openshift-image-registry/node-ca-cdxlq"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205338    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-host-var-lib-cni-bin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205367    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205396    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-kubelet-config\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205416    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-netns\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205419    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-run-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205431    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-registration-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205449    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205480    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-netns\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205493    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e73b381-beec-43fe-ba3d-460efc605f8b-kubelet-config\") pod \"global-pull-secret-syncer-pcvlc\" (UID: \"3e73b381-beec-43fe-ba3d-460efc605f8b\") " pod="kube-system/global-pull-secret-syncer-pcvlc"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205485    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-var-lib-kubelet\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205531    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-registration-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205536    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205586    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.205593    5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205580    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-var-lib-kubelet\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.205624    5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205628    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205665    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-socket-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205692    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205716    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-bin\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205763    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205787    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205812    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4738a53-824f-4a8b-ab86-b160b9f14c0e-hosts-file\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205838    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-systemd\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205855    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-socket-dir\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205862    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cnibin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205903    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-cnibin\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.205923    5074 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205960    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-cni-bin\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.205897    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-conf-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206006    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4738a53-824f-4a8b-ab86-b160b9f14c0e-hosts-file\") pod \"node-resolver-c99qk\" (UID: \"c4738a53-824f-4a8b-ab86-b160b9f14c0e\") " pod="openshift-dns/node-resolver-c99qk"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206024    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-modprobe-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206051    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-systemd\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206085    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206089    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1e7a5aa-87e3-4afd-ae59-801a559622f6-multus-conf-dir\") pod \"multus-jt2sr\" (UID: \"a1e7a5aa-87e3-4afd-ae59-801a559622f6\") " pod="openshift-multus/multus-jt2sr"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206053    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysconfig\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206177    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-etc-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206201    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75649c60-b6dd-4fe0-96f7-ff34c763de46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206206    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-modprobe-d\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.206181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206133    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206209    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-systemd-units\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206241    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysconfig\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206260    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-etc-openvswitch\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206274    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-sys-fs\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206292    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc8021d-2812-40f5-99ba-85e10f04ef4f-systemd-units\") pod \"ovnkube-node-ncc9n\" (UID: \"5bc8021d-2812-40f5-99ba-85e10f04ef4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206310    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cnibin\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206338    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206353    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63593602-b03b-47c5-9d74-a95ff2abfddb-sys-fs\") pod \"aws-ebs-csi-driver-node-jbbn4\" (UID: \"63593602-b03b-47c5-9d74-a95ff2abfddb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbbn4"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206369    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75649c60-b6dd-4fe0-96f7-ff34c763de46-crio-socket\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206398    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3bbd0ed-0bf6-401c-8ea6-d759c3909adc-cnibin\") pod \"multus-additional-cni-plugins-cqzg8\" (UID: \"a3bbd0ed-0bf6-401c-8ea6-d759c3909adc\") " pod="openshift-multus/multus-additional-cni-plugins-cqzg8"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206412    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75649c60-b6dd-4fe0-96f7-ff34c763de46-crio-socket\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206448    5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206470    5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206554    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:56.706535477 +0000 UTC m=+1.806135598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206593    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:56.70658413 +0000 UTC m=+1.806184255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206618    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:56.706608442 +0000 UTC m=+1.806208565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206639    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206665    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-conf\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206766    5074 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.206828    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:56.706811785 +0000 UTC m=+1.806411947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.206870    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0dae53-7734-414d-94ca-7bfd0bbc5d62-etc-sysctl-conf\") pod \"tuned-dxj77\" (UID: \"3a0dae53-7734-414d-94ca-7bfd0bbc5d62\") " pod="openshift-cluster-node-tuning-operator/tuned-dxj77"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.207024    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/698112a3-0157-43a7-9ea6-30e554d8be51-service-ca-bundle\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.207187    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cs8jk\" (UID: \"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.207304    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698112a3-0157-43a7-9ea6-30e554d8be51-metrics-certs\") pod \"router-default-5bb58b79c9-bksx8\" (UID: \"698112a3-0157-43a7-9ea6-30e554d8be51\") " pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.208269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.207609    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4741d3bf-14a8-4118-a61e-32dd236755dd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ngdxl\" (UID: \"4741d3bf-14a8-4118-a61e-32dd236755dd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.286059 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.285950    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:56.286335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.286314    5074 scope.go:122] "RemoveContainer" containerID="3ce0fa3e52d4ea9d6421653a85c47590d0b1e8314af6e1aaa0527f4c8122b495"
Nov 03 23:28:56.287437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.287413    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"
Nov 03 23:28:56.289838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.289817    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"
Nov 03 23:28:56.648767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.648671    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb58b79c9-bksx8"]
Nov 03 23:28:56.652423 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:56.652391    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698112a3_0157_43a7_9ea6_30e554d8be51.slice/crio-9c2efa6abf83caa0917655f4ab9803b8a6356acea1bfb2319796ff47944c2472 WatchSource:0}: Error finding container 9c2efa6abf83caa0917655f4ab9803b8a6356acea1bfb2319796ff47944c2472: Status 404 returned error can't find the container with id 9c2efa6abf83caa0917655f4ab9803b8a6356acea1bfb2319796ff47944c2472
Nov 03 23:28:56.674192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.674167    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl"]
Nov 03 23:28:56.682949 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:56.682922    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4741d3bf_14a8_4118_a61e_32dd236755dd.slice/crio-d189bc1be0b9ccda7604774b8c739ce46e0e1e69e4a472de63517832460d05c2 WatchSource:0}: Error finding container d189bc1be0b9ccda7604774b8c739ce46e0e1e69e4a472de63517832460d05c2: Status 404 returned error can't find the container with id d189bc1be0b9ccda7604774b8c739ce46e0e1e69e4a472de63517832460d05c2
Nov 03 23:28:56.685425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.685404    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:28:56.707414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.707329    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cs8jk"]
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.712182    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.712259    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.712314    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712342    5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712364    5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6768769b75-2ms22: secret "image-registry-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712387    5074 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712413    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls podName:01084ce5-4229-4c46-bb4e-9c6973a000e4 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.712393101 +0000 UTC m=+2.811993252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls") pod "image-registry-6768769b75-2ms22" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4") : secret "image-registry-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.712343    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712418    5074 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712430    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls podName:a32ab19d-6150-424c-8ce9-d3c8fcf7f967 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.712418721 +0000 UTC m=+2.812018844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls") pod "dns-default-4x94t" (UID: "a32ab19d-6150-424c-8ce9-d3c8fcf7f967") : secret "dns-default-metrics-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712453    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.712444758 +0000 UTC m=+2.812044932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712471    5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712485    5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b96cb8d6-ctr95: secret "image-registry-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.712526    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:56.712531    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd79947d_8f7a_4a12_8e8b_c8623c2e0e7d.slice/crio-1927bfcd9811d1f9dbe7e8b955557fb4f3fdc0655a219875441e3f2e276edfb3 WatchSource:0}: Error finding container 1927bfcd9811d1f9dbe7e8b955557fb4f3fdc0655a219875441e3f2e276edfb3: Status 404 returned error can't find the container with id 1927bfcd9811d1f9dbe7e8b955557fb4f3fdc0655a219875441e3f2e276edfb3
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712565    5074 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.712566    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712598    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.712584868 +0000 UTC m=+2.812185005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : secret "metrics-daemon-secret" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712625    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls podName:f74969b1-2e46-4e1c-ae7e-a56a2030b5b7 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.712615319 +0000 UTC m=+2.812215447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls") pod "image-registry-5b96cb8d6-ctr95" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7") : secret "image-registry-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712638    5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:56.712695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:56.712673    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:57.712662242 +0000 UTC m=+2.812262375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:56.913114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:56.913019    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:28:57.087921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.087885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb58b79c9-bksx8" event={"ID":"698112a3-0157-43a7-9ea6-30e554d8be51","Type":"ContainerStarted","Data":"3a36e24257879180660f0bddb37e6183532e3e1253671eccc432d763b4300511"}
Nov 03 23:28:57.088418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.087948    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb58b79c9-bksx8" event={"ID":"698112a3-0157-43a7-9ea6-30e554d8be51","Type":"ContainerStarted","Data":"9c2efa6abf83caa0917655f4ab9803b8a6356acea1bfb2319796ff47944c2472"}
Nov 03 23:28:57.090167 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.090141    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:28:57.090366 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.090237    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj" event={"ID":"e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb","Type":"ContainerStarted","Data":"2d4fd6d14220f92226f3d0079b74926f6ff5e915fc8033b1d3bbb3906c07a7d2"}
Nov 03 23:28:57.091614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.091585    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl" event={"ID":"4741d3bf-14a8-4118-a61e-32dd236755dd","Type":"ContainerStarted","Data":"d189bc1be0b9ccda7604774b8c739ce46e0e1e69e4a472de63517832460d05c2"}
Nov 03 23:28:57.092825 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.092792    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk" event={"ID":"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d","Type":"ContainerStarted","Data":"1927bfcd9811d1f9dbe7e8b955557fb4f3fdc0655a219875441e3f2e276edfb3"}
Nov 03 23:28:57.286841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.286805    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:57.289901 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.289870    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:57.723255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.723215    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:57.723466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.723287    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:57.723466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.723319    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:57.723466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.723381    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:57.723466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.723413    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:57.723696 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.723475    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:57.724237 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:57.723792    5074 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:57.724237 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:57.723872    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.723852223 +0000 UTC m=+4.823452364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:57.724237 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:57.723899    5074 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 23:28:57.724237 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:57.723966    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.723947988 +0000 UTC m=+4.823548112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : secret "metrics-daemon-secret" not found
Nov 03 23:28:57.724237 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:57.724039    5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:57.724237 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:57.724075    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:59.724064089 +0000 UTC m=+4.823664215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:57.726511 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.726489    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"image-registry-6768769b75-2ms22\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") " pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:57.727015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.726996    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a32ab19d-6150-424c-8ce9-d3c8fcf7f967-metrics-tls\") pod \"dns-default-4x94t\" (UID: \"a32ab19d-6150-424c-8ce9-d3c8fcf7f967\") " pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:57.727234 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.727214    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"image-registry-5b96cb8d6-ctr95\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") " pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:57.788644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.788608    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:28:57.799816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.799792    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:57.800879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:57.800853    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4x94t"
Nov 03 23:28:58.056223 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:58.056150    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:58.057960 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:58.057927    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5bb58b79c9-bksx8"
Nov 03 23:28:58.136193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:58.136152    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6768769b75-2ms22"]
Nov 03 23:28:58.140757 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:58.140703    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01084ce5_4229_4c46_bb4e_9c6973a000e4.slice/crio-efee1db92f8fddc6eebe1c2f3d9bfb3f028dc676b9d5bdf4d577fa3ba7664f2b WatchSource:0}: Error finding container efee1db92f8fddc6eebe1c2f3d9bfb3f028dc676b9d5bdf4d577fa3ba7664f2b: Status 404 returned error can't find the container with id efee1db92f8fddc6eebe1c2f3d9bfb3f028dc676b9d5bdf4d577fa3ba7664f2b
Nov 03 23:28:58.171100 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:58.171069    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4x94t"]
Nov 03 23:28:58.175210 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:28:58.175172    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32ab19d_6150_424c_8ce9_d3c8fcf7f967.slice/crio-f3e20773869fcfa7aa432c5bc8d8b591e2a2c2afaf6b57ef63bbe5e3f3879bd6 WatchSource:0}: Error finding container f3e20773869fcfa7aa432c5bc8d8b591e2a2c2afaf6b57ef63bbe5e3f3879bd6: Status 404 returned error can't find the container with id f3e20773869fcfa7aa432c5bc8d8b591e2a2c2afaf6b57ef63bbe5e3f3879bd6
Nov 03 23:28:58.177248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:58.177165    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b96cb8d6-ctr95"]
Nov 03 23:28:59.104383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.104340    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cs8jk" event={"ID":"dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d","Type":"ContainerStarted","Data":"d1c587f4a3eee9d5179fd919c2dab08f8477f7f299708425cc6c91bf1e87699f"}
Nov 03 23:28:59.106292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.106261    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6768769b75-2ms22" event={"ID":"01084ce5-4229-4c46-bb4e-9c6973a000e4","Type":"ContainerStarted","Data":"2ed59dec574492f59aff5b84c26161d9cc98ea261c6d4cc92e8d3953589309d2"}
Nov 03 23:28:59.106423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.106315    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6768769b75-2ms22" event={"ID":"01084ce5-4229-4c46-bb4e-9c6973a000e4","Type":"ContainerStarted","Data":"efee1db92f8fddc6eebe1c2f3d9bfb3f028dc676b9d5bdf4d577fa3ba7664f2b"}
Nov 03 23:28:59.107622 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.107594    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4x94t" event={"ID":"a32ab19d-6150-424c-8ce9-d3c8fcf7f967","Type":"ContainerStarted","Data":"f3e20773869fcfa7aa432c5bc8d8b591e2a2c2afaf6b57ef63bbe5e3f3879bd6"}
Nov 03 23:28:59.109251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.109226    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ngdxl" event={"ID":"4741d3bf-14a8-4118-a61e-32dd236755dd","Type":"ContainerStarted","Data":"43e5f512a16025a34c15b30328ec8879207ce3a67a592402eab3f2fe9d9993b8"}
Nov 03 23:28:59.111086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.111059    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95" event={"ID":"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7","Type":"ContainerStarted","Data":"14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655"}
Nov 03 23:28:59.111199 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.111096    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95" event={"ID":"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7","Type":"ContainerStarted","Data":"f68e05fb4588f7170ca9eac33ba7c9eb52fcbef0e3b14cc89c1e395aec111472"}
Nov 03 23:28:59.413360 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.413287    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:28:59.555275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.555245    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:28:59.748470 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.748433    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:28:59.748644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.748499    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:28:59.748644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:28:59.748541    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:28:59.748731 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:59.748683    5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:59.748801 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:59.748773    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls podName:75649c60-b6dd-4fe0-96f7-ff34c763de46 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:03.748734577 +0000 UTC m=+8.848334711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzv94" (UID: "75649c60-b6dd-4fe0-96f7-ff34c763de46") : secret "insights-runtime-extractor-tls" not found
Nov 03 23:28:59.749141 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:59.749124    5074 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 23:28:59.749190 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:59.749172    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls podName:40135659-f74d-4307-8bb3-574411040f5a nodeName:}" failed. No retries permitted until 2026-11-03 23:29:03.74915999 +0000 UTC m=+8.848760119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5s52h" (UID: "40135659-f74d-4307-8bb3-574411040f5a") : secret "samples-operator-tls" not found
Nov 03 23:28:59.749245 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:59.749213    5074 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 23:28:59.749245 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:28:59.749240    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs podName:1244c649-9fe7-40c5-9b55-443a2529c710 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:03.749233578 +0000 UTC m=+8.848833700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs") pod "network-metrics-daemon-6n2fd" (UID: "1244c649-9fe7-40c5-9b55-443a2529c710") : secret "metrics-daemon-secret" not found
Nov 03 23:29:00.116910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.116881    5074 generic.go:356] "Generic (PLEG): container finished" podID="13b0dc51-f5cc-4afd-821f-c3097be21e80" containerID="3239e2bd93fc77aa944e825b031ff9ad3e2af27dbc65fc0f50f3fc8a87c5de10" exitCode=0
Nov 03 23:29:00.117047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.116966    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj" event={"ID":"13b0dc51-f5cc-4afd-821f-c3097be21e80","Type":"ContainerDied","Data":"3239e2bd93fc77aa944e825b031ff9ad3e2af27dbc65fc0f50f3fc8a87c5de10"}
Nov 03 23:29:00.117371 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.117353    5074 scope.go:122] "RemoveContainer" containerID="3239e2bd93fc77aa944e825b031ff9ad3e2af27dbc65fc0f50f3fc8a87c5de10"
Nov 03 23:29:00.118599 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.118568    5074 generic.go:356] "Generic (PLEG): container finished" podID="69d478d4-7d64-4a68-823e-29c6b40b4a94" containerID="ebdfd07d9283bb1e23b0f5a8471d3f2072f1fcfddcdb641d6006184613274af7" exitCode=0
Nov 03 23:29:00.118705 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.118678    5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:29:00.118705 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.118699    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7869h" event={"ID":"69d478d4-7d64-4a68-823e-29c6b40b4a94","Type":"ContainerDied","Data":"ebdfd07d9283bb1e23b0f5a8471d3f2072f1fcfddcdb641d6006184613274af7"}
Nov 03 23:29:00.119128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:00.119111    5074 scope.go:122] "RemoveContainer" containerID="ebdfd07d9283bb1e23b0f5a8471d3f2072f1fcfddcdb641d6006184613274af7"
Nov 03 23:29:01.126598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.126563    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4x94t" event={"ID":"a32ab19d-6150-424c-8ce9-d3c8fcf7f967","Type":"ContainerStarted","Data":"a048f61348665d50dcdf7fab2d5e17ebdb738c77e40ceec6b0ba7a9ea76e8d14"}
Nov 03 23:29:01.127126 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.126616    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4x94t" event={"ID":"a32ab19d-6150-424c-8ce9-d3c8fcf7f967","Type":"ContainerStarted","Data":"a9efce5fcacf42f0ad2c83b5f8bbd1177265669ae58174cb2f44d1627b3af07d"}
Nov 03 23:29:01.127126 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.126677    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4x94t"
Nov 03 23:29:01.128367 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.128343    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj" event={"ID":"13b0dc51-f5cc-4afd-821f-c3097be21e80","Type":"ContainerStarted","Data":"66d0dab7789bea4acca21bbce410e52a3ae57700bd378b18206e41d2e73db7e5"}
Nov 03 23:29:01.129660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.129640    5074 generic.go:356] "Generic (PLEG): container finished" podID="3cb8c88a-1985-4ca8-a11f-7ab86cd87634" containerID="43969488b58c725d8d66e26a00d77105d360d9d9c988db90467f3b8ee67291c3" exitCode=0
Nov 03 23:29:01.129757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.129709    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc" event={"ID":"3cb8c88a-1985-4ca8-a11f-7ab86cd87634","Type":"ContainerDied","Data":"43969488b58c725d8d66e26a00d77105d360d9d9c988db90467f3b8ee67291c3"}
Nov 03 23:29:01.130042 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.130025    5074 scope.go:122] "RemoveContainer" containerID="43969488b58c725d8d66e26a00d77105d360d9d9c988db90467f3b8ee67291c3"
Nov 03 23:29:01.131517 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:01.131492    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7869h" event={"ID":"69d478d4-7d64-4a68-823e-29c6b40b4a94","Type":"ContainerStarted","Data":"8dcfd28d80b2dcbacaac31e1463ba39a257d3de0b74ce8ef5a9d64fec8828e2c"}
Nov 03 23:29:02.137781 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:02.137729    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8smkc" event={"ID":"3cb8c88a-1985-4ca8-a11f-7ab86cd87634","Type":"ContainerStarted","Data":"18f74ed8482aca495c1d50fc552aebcf35305b33f6d04bb3bf320aa5a761d12c"}
Nov 03 23:29:03.789279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.789239    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:29:03.789668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.789291    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:29:03.789668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.789315    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:29:03.791799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.791775    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75649c60-b6dd-4fe0-96f7-ff34c763de46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzv94\" (UID: \"75649c60-b6dd-4fe0-96f7-ff34c763de46\") " pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:29:03.791923 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.791877    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1244c649-9fe7-40c5-9b55-443a2529c710-metrics-certs\") pod \"network-metrics-daemon-6n2fd\" (UID: \"1244c649-9fe7-40c5-9b55-443a2529c710\") " pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:29:03.791996 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.791982    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6n2fd"
Nov 03 23:29:03.792029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.792011    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40135659-f74d-4307-8bb3-574411040f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5s52h\" (UID: \"40135659-f74d-4307-8bb3-574411040f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:29:03.802780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.802761    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzv94"
Nov 03 23:29:03.812059 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.812040    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:29:03.812181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.812168    5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:29:03.816484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:03.816467    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6sdkj"
Nov 03 23:29:04.069830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.069806    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6n2fd"]
Nov 03 23:29:04.072333 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:04.072308    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1244c649_9fe7_40c5_9b55_443a2529c710.slice/crio-0e1049e77cec070f7c21426766acc94f4fcb1ffb466f2bdacd202d2a8336635e WatchSource:0}: Error finding container 0e1049e77cec070f7c21426766acc94f4fcb1ffb466f2bdacd202d2a8336635e: Status 404 returned error can't find the container with id 0e1049e77cec070f7c21426766acc94f4fcb1ffb466f2bdacd202d2a8336635e
Nov 03 23:29:04.081304 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.081282    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dzv94"]
Nov 03 23:29:04.083647 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:04.083626    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75649c60_b6dd_4fe0_96f7_ff34c763de46.slice/crio-84444c003e8dc2c702421d0b4523e556cd1547b57b10bfefa0c819a8564248dd WatchSource:0}: Error finding container 84444c003e8dc2c702421d0b4523e556cd1547b57b10bfefa0c819a8564248dd: Status 404 returned error can't find the container with id 84444c003e8dc2c702421d0b4523e556cd1547b57b10bfefa0c819a8564248dd
Nov 03 23:29:04.087734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.087720    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"
Nov 03 23:29:04.149343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.149311    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6n2fd" event={"ID":"1244c649-9fe7-40c5-9b55-443a2529c710","Type":"ContainerStarted","Data":"0e1049e77cec070f7c21426766acc94f4fcb1ffb466f2bdacd202d2a8336635e"}
Nov 03 23:29:04.151601 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.151577    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzv94" event={"ID":"75649c60-b6dd-4fe0-96f7-ff34c763de46","Type":"ContainerStarted","Data":"84444c003e8dc2c702421d0b4523e556cd1547b57b10bfefa0c819a8564248dd"}
Nov 03 23:29:04.361574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.361509    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h"]
Nov 03 23:29:04.632259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.632182    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:29:04.632473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:04.632449    5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:29:05.159775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:05.157465    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzv94" event={"ID":"75649c60-b6dd-4fe0-96f7-ff34c763de46","Type":"ContainerStarted","Data":"87de398a3b22e05cd3fdfdfd7f8b660f5302dc7916e60721281476f1befd132d"}
Nov 03 23:29:05.159775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:05.159433    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h" event={"ID":"40135659-f74d-4307-8bb3-574411040f5a","Type":"ContainerStarted","Data":"44a8c08b74aea7bb8ddd3479e18aaf63f13ece96ac8c1aa52c29e7b212789b4a"}
Nov 03 23:29:06.165586 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:06.165549    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzv94" event={"ID":"75649c60-b6dd-4fe0-96f7-ff34c763de46","Type":"ContainerStarted","Data":"f9e959376e173aa3fe3245f25e57a5017791462bdcfd06ee4a9e25b60e36ef00"}
Nov 03 23:29:06.167221 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:06.167194    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6n2fd" event={"ID":"1244c649-9fe7-40c5-9b55-443a2529c710","Type":"ContainerStarted","Data":"6b5228f902594e39ee3e47f41c991e23ce84eda298a759911753851bda780247"}
Nov 03 23:29:06.167312 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:06.167240    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6n2fd" event={"ID":"1244c649-9fe7-40c5-9b55-443a2529c710","Type":"ContainerStarted","Data":"0c8866c86390b0648c247becf88592b75d065eed105ad469281fce54f3b917fd"}
Nov 03 23:29:06.678723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:06.678691    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:29:07.175706 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:07.175669    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzv94" event={"ID":"75649c60-b6dd-4fe0-96f7-ff34c763de46","Type":"ContainerStarted","Data":"b1c5a4421bf8f1cd1443e08a72926f723803a40df3fb2ad8243990b39d045430"}
Nov 03 23:29:11.140834 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:11.140805    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4x94t"
Nov 03 23:29:12.067603 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.067570    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-w5bsk-9tpvz"]
Nov 03 23:29:12.070963 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.070944    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.085627 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.085605    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w5bsk-9tpvz"]
Nov 03 23:29:12.154363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.154335    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"]
Nov 03 23:29:12.160619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.160583    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nsh2h"]
Nov 03 23:29:12.160773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.160682    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:12.164244 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.164219    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.164504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.164473    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.164693 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.164674    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxth6\" (UniqueName: \"kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.164808 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.164708    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.164890 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.164869    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.168254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.168230    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6768769b75-2ms22"]
Nov 03 23:29:12.169117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.169095    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-55857b5d96-gc7z7"]
Nov 03 23:29:12.172518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.172502    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.192389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.192368    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 23:29:12.192495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.192456    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 23:29:12.192495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.192468    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 23:29:12.192660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.192631    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 23:29:12.193583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.193562    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 23:29:12.193787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.193733    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 23:29:12.193856 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.193789    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 23:29:12.194418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.194399    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dmzm2\""
Nov 03 23:29:12.194698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.194680    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 23:29:12.194841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.194815    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-brfjf\""
Nov 03 23:29:12.194929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.194844    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hpfml\""
Nov 03 23:29:12.198505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.198482    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h" event={"ID":"40135659-f74d-4307-8bb3-574411040f5a","Type":"ContainerStarted","Data":"cf7af2d74a8939c4850309bae6873f53ad268a01523d58d3c44b9ba8a8557f4b"}
Nov 03 23:29:12.198586 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.198525    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5s52h" event={"ID":"40135659-f74d-4307-8bb3-574411040f5a","Type":"ContainerStarted","Data":"eee62da1c2c32b82b2bb49b4ed3d41f38f3fe16ea22da69f809fdbb181a6beea"}
Nov 03 23:29:12.206873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.206851    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nsh2h"]
Nov 03 23:29:12.207687 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.207665    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"]
Nov 03 23:29:12.242503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.242477    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55857b5d96-gc7z7"]
Nov 03 23:29:12.263249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.263218    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-795f84c5c9-ktnfg"]
Nov 03 23:29:12.265841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.265818    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxth6\" (UniqueName: \"kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.265971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.265856    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.265971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.265886    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.265971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.265942    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrpc\" (UniqueName: \"kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.266130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.265988    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.266130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266023    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.266130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266055    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.266130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266071    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdp2\" (UniqueName: \"kubernetes.io/projected/001ac09b-53c8-4437-bc59-d1785cc36723-kube-api-access-4sdp2\") pod \"downloads-6dd789c8b7-nsh2h\" (UID: \"001ac09b-53c8-4437-bc59-d1785cc36723\") " pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.266272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266159    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.266272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266206    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.266272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266223    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/001ac09b-53c8-4437-bc59-d1785cc36723-tmp\") pod \"downloads-6dd789c8b7-nsh2h\" (UID: \"001ac09b-53c8-4437-bc59-d1785cc36723\") " pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.266403 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266381    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1848d733-9923-43d8-8caa-11fb3d30cb24-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qtr79\" (UID: \"1848d733-9923-43d8-8caa-11fb3d30cb24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:12.266458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266421    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.266458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266446    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.266532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266459    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.266659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266637    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.266700 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.266643    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.267075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.267055    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.289949 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.289924    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795f84c5c9-ktnfg"]
Nov 03 23:29:12.291318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.291296    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxth6\" (UniqueName: \"kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6\") pod \"periodic-gathering-w5bsk-9tpvz\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") " pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.367723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367647    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.367723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367693    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5564e75b-0171-4100-ad93-6f6e6089f0e0-installation-pull-secrets\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.367723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367712    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367729    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdp2\" (UniqueName: \"kubernetes.io/projected/001ac09b-53c8-4437-bc59-d1785cc36723-kube-api-access-4sdp2\") pod \"downloads-6dd789c8b7-nsh2h\" (UID: \"001ac09b-53c8-4437-bc59-d1785cc36723\") " pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.368018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367761    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5564e75b-0171-4100-ad93-6f6e6089f0e0-registry-certificates\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.368018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367786    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-bound-sa-token\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.368018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367931    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-registry-tls\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.368018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.367994    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368075    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/001ac09b-53c8-4437-bc59-d1785cc36723-tmp\") pod \"downloads-6dd789c8b7-nsh2h\" (UID: \"001ac09b-53c8-4437-bc59-d1785cc36723\") " pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.368267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368123    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5564e75b-0171-4100-ad93-6f6e6089f0e0-image-registry-private-configuration\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.368267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368159    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1848d733-9923-43d8-8caa-11fb3d30cb24-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qtr79\" (UID: \"1848d733-9923-43d8-8caa-11fb3d30cb24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:12.368464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368436    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368438    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/001ac09b-53c8-4437-bc59-d1785cc36723-tmp\") pod \"downloads-6dd789c8b7-nsh2h\" (UID: \"001ac09b-53c8-4437-bc59-d1785cc36723\") " pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.368633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368518    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzpkm\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-kube-api-access-pzpkm\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.368633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368545    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368556    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5564e75b-0171-4100-ad93-6f6e6089f0e0-ca-trust-extracted\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.368821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368641    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368691    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368718    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrpc\" (UniqueName: \"kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.368821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.368767    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5564e75b-0171-4100-ad93-6f6e6089f0e0-trusted-ca\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.369563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.369052    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.369563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.369378    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.369563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.369521    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.370659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.370640    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.370806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.370786    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1848d733-9923-43d8-8caa-11fb3d30cb24-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qtr79\" (UID: \"1848d733-9923-43d8-8caa-11fb3d30cb24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:12.370848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.370813    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.383225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.383205    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:29:12.385071 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.385050    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrpc\" (UniqueName: \"kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc\") pod \"console-55857b5d96-gc7z7\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") " pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.387830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.387807    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdp2\" (UniqueName: \"kubernetes.io/projected/001ac09b-53c8-4437-bc59-d1785cc36723-kube-api-access-4sdp2\") pod \"downloads-6dd789c8b7-nsh2h\" (UID: \"001ac09b-53c8-4437-bc59-d1785cc36723\") " pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.470036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470006    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzpkm\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-kube-api-access-pzpkm\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470042    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5564e75b-0171-4100-ad93-6f6e6089f0e0-ca-trust-extracted\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470070    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5564e75b-0171-4100-ad93-6f6e6089f0e0-trusted-ca\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470094    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5564e75b-0171-4100-ad93-6f6e6089f0e0-installation-pull-secrets\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470111    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5564e75b-0171-4100-ad93-6f6e6089f0e0-registry-certificates\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470125    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-bound-sa-token\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470146    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-registry-tls\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470184    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5564e75b-0171-4100-ad93-6f6e6089f0e0-image-registry-private-configuration\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.470891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.470861    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5564e75b-0171-4100-ad93-6f6e6089f0e0-ca-trust-extracted\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.471281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.471260    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5564e75b-0171-4100-ad93-6f6e6089f0e0-registry-certificates\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.471569 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.471548    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5564e75b-0171-4100-ad93-6f6e6089f0e0-trusted-ca\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.472828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.472811    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-registry-tls\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.473514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.473495    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5564e75b-0171-4100-ad93-6f6e6089f0e0-image-registry-private-configuration\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.473613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.473594    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5564e75b-0171-4100-ad93-6f6e6089f0e0-installation-pull-secrets\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.476473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.476457    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:12.485159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.485143    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:29:12.487362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.487339    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-bound-sa-token\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.492256 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.492236    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:12.497949 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.497924    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzpkm\" (UniqueName: \"kubernetes.io/projected/5564e75b-0171-4100-ad93-6f6e6089f0e0-kube-api-access-pzpkm\") pod \"image-registry-795f84c5c9-ktnfg\" (UID: \"5564e75b-0171-4100-ad93-6f6e6089f0e0\") " pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.577785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.577723    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:12.681646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.680065    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w5bsk-9tpvz"]
Nov 03 23:29:12.683055 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:12.683024    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b959e5_0c65_457f_8f9b_d386854c3642.slice/crio-b9a0f50267f9d509d8de224f86978079214206f212d631a2e5a10149c43256f4 WatchSource:0}: Error finding container b9a0f50267f9d509d8de224f86978079214206f212d631a2e5a10149c43256f4: Status 404 returned error can't find the container with id b9a0f50267f9d509d8de224f86978079214206f212d631a2e5a10149c43256f4
Nov 03 23:29:12.756571 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.756542    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"]
Nov 03 23:29:12.759992 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:12.759966    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1848d733_9923_43d8_8caa_11fb3d30cb24.slice/crio-d9f4ad0c3ba07ddb132260173662bb89fac278f998977298aa620dc07f3ca9b3 WatchSource:0}: Error finding container d9f4ad0c3ba07ddb132260173662bb89fac278f998977298aa620dc07f3ca9b3: Status 404 returned error can't find the container with id d9f4ad0c3ba07ddb132260173662bb89fac278f998977298aa620dc07f3ca9b3
Nov 03 23:29:12.782196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.782165    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nsh2h"]
Nov 03 23:29:12.785533 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:12.785502    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001ac09b_53c8_4437_bc59_d1785cc36723.slice/crio-b9b381f01e72bf27d9aa9771c189917d623176af6d82f68bdb3e2cf2e1687522 WatchSource:0}: Error finding container b9b381f01e72bf27d9aa9771c189917d623176af6d82f68bdb3e2cf2e1687522: Status 404 returned error can't find the container with id b9b381f01e72bf27d9aa9771c189917d623176af6d82f68bdb3e2cf2e1687522
Nov 03 23:29:12.790644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.790619    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55857b5d96-gc7z7"]
Nov 03 23:29:12.797070 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:12.797044    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e412ff_1d0b_4236_8686_401a852f8e3e.slice/crio-4b47d257d358a08b404232424f7bf1313c7b9f860b7c8f7d234961a38d1bca77 WatchSource:0}: Error finding container 4b47d257d358a08b404232424f7bf1313c7b9f860b7c8f7d234961a38d1bca77: Status 404 returned error can't find the container with id 4b47d257d358a08b404232424f7bf1313c7b9f860b7c8f7d234961a38d1bca77
Nov 03 23:29:12.862153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:12.862045    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795f84c5c9-ktnfg"]
Nov 03 23:29:12.864670 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:12.864648    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5564e75b_0171_4100_ad93_6f6e6089f0e0.slice/crio-581c5c95a41a200fd3d92bce65e804db2361da75dd989d8ebb6ee734b86b26e8 WatchSource:0}: Error finding container 581c5c95a41a200fd3d92bce65e804db2361da75dd989d8ebb6ee734b86b26e8: Status 404 returned error can't find the container with id 581c5c95a41a200fd3d92bce65e804db2361da75dd989d8ebb6ee734b86b26e8
Nov 03 23:29:13.208666 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.208631    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg" event={"ID":"5564e75b-0171-4100-ad93-6f6e6089f0e0","Type":"ContainerStarted","Data":"8a2c6d02a5d35fe6d4a2fdeed1e6c2d9180eb40e58a80b58d7f6bd7a870e02f2"}
Nov 03 23:29:13.209174 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.208708    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg" event={"ID":"5564e75b-0171-4100-ad93-6f6e6089f0e0","Type":"ContainerStarted","Data":"581c5c95a41a200fd3d92bce65e804db2361da75dd989d8ebb6ee734b86b26e8"}
Nov 03 23:29:13.209174 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.208765    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:13.210551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.210521    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55857b5d96-gc7z7" event={"ID":"18e412ff-1d0b-4236-8686-401a852f8e3e","Type":"ContainerStarted","Data":"4b47d257d358a08b404232424f7bf1313c7b9f860b7c8f7d234961a38d1bca77"}
Nov 03 23:29:13.211893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.211867    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nsh2h" event={"ID":"001ac09b-53c8-4437-bc59-d1785cc36723","Type":"ContainerStarted","Data":"b9b381f01e72bf27d9aa9771c189917d623176af6d82f68bdb3e2cf2e1687522"}
Nov 03 23:29:13.213773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.213732    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz" event={"ID":"96b959e5-0c65-457f-8f9b-d386854c3642","Type":"ContainerStarted","Data":"0134582ffee02c4be29da39bb8c2c00c389a70b644f574a87f829ed967feb3f2"}
Nov 03 23:29:13.213870 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.213788    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz" event={"ID":"96b959e5-0c65-457f-8f9b-d386854c3642","Type":"ContainerStarted","Data":"b9a0f50267f9d509d8de224f86978079214206f212d631a2e5a10149c43256f4"}
Nov 03 23:29:13.215127 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.215097    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79" event={"ID":"1848d733-9923-43d8-8caa-11fb3d30cb24","Type":"ContainerStarted","Data":"d9f4ad0c3ba07ddb132260173662bb89fac278f998977298aa620dc07f3ca9b3"}
Nov 03 23:29:13.244634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.244587    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg" podStartSLOduration=1.244573316 podStartE2EDuration="1.244573316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:13.244169186 +0000 UTC m=+18.343769345" watchObservedRunningTime="2026-11-03 23:29:13.244573316 +0000 UTC m=+18.344173454"
Nov 03 23:29:13.283820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.283765    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz" podStartSLOduration=1.283730316 podStartE2EDuration="1.283730316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:13.282857294 +0000 UTC m=+18.382457438" watchObservedRunningTime="2026-11-03 23:29:13.283730316 +0000 UTC m=+18.383330458"
Nov 03 23:29:13.524212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.524185    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4x94t_a32ab19d-6150-424c-8ce9-d3c8fcf7f967/dns/0.log"
Nov 03 23:29:13.723903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:13.723869    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4x94t_a32ab19d-6150-424c-8ce9-d3c8fcf7f967/kube-rbac-proxy/0.log"
Nov 03 23:29:14.123569 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:14.123541    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c99qk_c4738a53-824f-4a8b-ab86-b160b9f14c0e/dns-node-resolver/0.log"
Nov 03 23:29:14.525577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:14.525550    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b96cb8d6-ctr95_f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/registry/0.log"
Nov 03 23:29:14.725824 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:14.725724    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6768769b75-2ms22_01084ce5-4229-4c46-bb4e-9c6973a000e4/registry/0.log"
Nov 03 23:29:14.925346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:14.925275    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-795f84c5c9-ktnfg_5564e75b-0171-4100-ad93-6f6e6089f0e0/registry/0.log"
Nov 03 23:29:15.229233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.229188    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79" event={"ID":"1848d733-9923-43d8-8caa-11fb3d30cb24","Type":"ContainerStarted","Data":"555e7c4543c7b338616ffeb49cdd2aa8d2c8a9dba3e0a8f94413314c8b95abc2"}
Nov 03 23:29:15.229468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.229449    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:15.246945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.246910    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79"
Nov 03 23:29:15.265044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.264985    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qtr79" podStartSLOduration=1.770337675 podStartE2EDuration="3.264967799s" totalImagesPullingTime="1.494630124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:15.262825754 +0000 UTC m=+20.362425900" watchObservedRunningTime="2026-11-03 23:29:15.264967799 +0000 UTC m=+20.364567942"
Nov 03 23:29:15.323368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.323342    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdxlq_cd9bbc11-d972-4662-bb0d-b1d8744673c9/node-ca/0.log"
Nov 03 23:29:15.724295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.724254    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb58b79c9-bksx8_698112a3-0157-43a7-9ea6-30e554d8be51/router/0.log"
Nov 03 23:29:15.924515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:15.924484    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6mthf_3b107666-eaaa-4d08-839b-d3e91dd5868b/serve-healthcheck-canary/0.log"
Nov 03 23:29:16.234707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:16.234626    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55857b5d96-gc7z7" event={"ID":"18e412ff-1d0b-4236-8686-401a852f8e3e","Type":"ContainerStarted","Data":"9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de"}
Nov 03 23:29:16.263294 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:16.263204    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-55857b5d96-gc7z7" podStartSLOduration=1.117430371 podStartE2EDuration="4.263165366s" totalImagesPullingTime="3.145734995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:16.262047039 +0000 UTC m=+21.361647184" watchObservedRunningTime="2026-11-03 23:29:16.263165366 +0000 UTC m=+21.362765501"
Nov 03 23:29:16.572211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:16.572136    5074 patch_prober.go:29] interesting pod/image-registry-5b96cb8d6-ctr95 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 23:29:16.572211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:16.572189    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95" podUID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:29:19.528686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:19.528656    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b9jnr"
Nov 03 23:29:19.690107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:19.690078    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncc9n"
Nov 03 23:29:21.136459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:21.136426    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:29:21.943792 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:21.943732    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-czggl"]
Nov 03 23:29:21.988642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:21.988617    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:21.993405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:21.993382    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 23:29:21.995678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:21.995390    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 23:29:22.017305 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.017277    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 23:29:22.017396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.017307    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fw76h\""
Nov 03 23:29:22.017396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.017351    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 23:29:22.065076 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065049    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-root\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065122    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-textfile\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065159    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065263 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065244    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-tls\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065314 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065301    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-sys\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065352 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065337    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065424 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065411    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttzh\" (UniqueName: \"kubernetes.io/projected/b78183df-7e18-4a6d-b121-b25605f9d0e7-kube-api-access-6ttzh\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065441    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-wtmp\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.065492 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.065481    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b78183df-7e18-4a6d-b121-b25605f9d0e7-metrics-client-ca\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166097    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-sys\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166149    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166195    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttzh\" (UniqueName: \"kubernetes.io/projected/b78183df-7e18-4a6d-b121-b25605f9d0e7-kube-api-access-6ttzh\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166207    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-sys\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166232    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-wtmp\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166253    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b78183df-7e18-4a6d-b121-b25605f9d0e7-metrics-client-ca\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166287    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-root\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166306    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-textfile\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166336    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166404    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-root\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166446    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-wtmp\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.166522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166450    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-tls\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.167025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166686    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-textfile\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.167025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166887    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.167025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.166932    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b78183df-7e18-4a6d-b121-b25605f9d0e7-metrics-client-ca\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.168678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.168659    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-tls\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.168788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.168666    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b78183df-7e18-4a6d-b121-b25605f9d0e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.174648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.174628    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:29:22.184135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.184116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttzh\" (UniqueName: \"kubernetes.io/projected/b78183df-7e18-4a6d-b121-b25605f9d0e7-kube-api-access-6ttzh\") pod \"node-exporter-czggl\" (UID: \"b78183df-7e18-4a6d-b121-b25605f9d0e7\") " pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.346291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.346217    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-czggl"
Nov 03 23:29:22.378551 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:29:22.378520    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78183df_7e18_4a6d_b121_b25605f9d0e7.slice/crio-ed9191bac97f36538ddd7fe69421b32f3b50219c0250769c506d11106502370a WatchSource:0}: Error finding container ed9191bac97f36538ddd7fe69421b32f3b50219c0250769c506d11106502370a: Status 404 returned error can't find the container with id ed9191bac97f36538ddd7fe69421b32f3b50219c0250769c506d11106502370a
Nov 03 23:29:22.493356 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.493324    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:22.493511 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.493396    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:22.498362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:22.498339    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:23.267638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:23.267605    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czggl" event={"ID":"b78183df-7e18-4a6d-b121-b25605f9d0e7","Type":"ContainerStarted","Data":"ed9191bac97f36538ddd7fe69421b32f3b50219c0250769c506d11106502370a"}
Nov 03 23:29:23.271518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:23.271493    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:25.277296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:25.277259    5074 generic.go:356] "Generic (PLEG): container finished" podID="b78183df-7e18-4a6d-b121-b25605f9d0e7" containerID="391c1546382f1fa352471f23f38e86014ba222e61bd3e8a5440f4fcadefb8417" exitCode=0
Nov 03 23:29:25.277705 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:25.277337    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czggl" event={"ID":"b78183df-7e18-4a6d-b121-b25605f9d0e7","Type":"ContainerDied","Data":"391c1546382f1fa352471f23f38e86014ba222e61bd3e8a5440f4fcadefb8417"}
Nov 03 23:29:26.284131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:26.284099    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czggl" event={"ID":"b78183df-7e18-4a6d-b121-b25605f9d0e7","Type":"ContainerStarted","Data":"c43798ea1a974dd75d352ce97b531a0de7afc1771c7ec85a00b7977f23a6540a"}
Nov 03 23:29:26.284503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:26.284153    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-czggl" event={"ID":"b78183df-7e18-4a6d-b121-b25605f9d0e7","Type":"ContainerStarted","Data":"6eabf91e7fbad3dd5708a41cae5db66059a03a173159384156544934ce55dd2e"}
Nov 03 23:29:26.318433 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:26.318381    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-czggl" podStartSLOduration=3.092155751 podStartE2EDuration="5.318367998s" totalImagesPullingTime="2.145108898s" totalInitContainerRuntime="81.103349ms" isStatefulPod=true podCreationTimestamp="2026-11-03 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:26.317211721 +0000 UTC m=+31.416811866" watchObservedRunningTime="2026-11-03 23:29:26.318367998 +0000 UTC m=+31.417968142"
Nov 03 23:29:28.021138 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:29:28.021104    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:29:28.481725 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:28.481692    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55857b5d96-gc7z7"]
Nov 03 23:29:29.416020 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:29.415991    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7cjjg"
Nov 03 23:29:34.224529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:34.224503    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-795f84c5c9-ktnfg"
Nov 03 23:29:34.303031 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:34.302989    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b96cb8d6-ctr95"]
Nov 03 23:29:37.189789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.189648    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6768769b75-2ms22" podUID="01084ce5-4229-4c46-bb4e-9c6973a000e4" containerName="registry" containerID="cri-o://2ed59dec574492f59aff5b84c26161d9cc98ea261c6d4cc92e8d3953589309d2" gracePeriod=30
Nov 03 23:29:37.337894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.337861    5074 generic.go:356] "Generic (PLEG): container finished" podID="01084ce5-4229-4c46-bb4e-9c6973a000e4" containerID="2ed59dec574492f59aff5b84c26161d9cc98ea261c6d4cc92e8d3953589309d2" exitCode=0
Nov 03 23:29:37.338079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.337929    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6768769b75-2ms22" event={"ID":"01084ce5-4229-4c46-bb4e-9c6973a000e4","Type":"ContainerDied","Data":"2ed59dec574492f59aff5b84c26161d9cc98ea261c6d4cc92e8d3953589309d2"}
Nov 03 23:29:37.433096 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.433071    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:29:37.526245 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526174    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526245 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526217    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526260    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526279    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526306    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526371    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526432    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526523    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") pod \"01084ce5-4229-4c46-bb4e-9c6973a000e4\" (UID: \"01084ce5-4229-4c46-bb4e-9c6973a000e4\") "
Nov 03 23:29:37.526854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526779    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:37.526854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.526846    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:37.528828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.528768    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:37.528828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.528782    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:29:37.529001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.528923    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:29:37.529062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.529018    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "kube-api-access-m9gr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:37.529140 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.529118    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:37.533598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.533567    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted" pod "01084ce5-4229-4c46-bb4e-9c6973a000e4" (UID: "01084ce5-4229-4c46-bb4e-9c6973a000e4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:29:37.627195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627168    5074 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-certificates\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627193    5074 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-registry-tls\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627205    5074 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-installation-pull-secrets\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627215    5074 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01084ce5-4229-4c46-bb4e-9c6973a000e4-image-registry-private-configuration\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627224    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gr2\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-kube-api-access-m9gr2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627233    5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01084ce5-4229-4c46-bb4e-9c6973a000e4-trusted-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627241    5074 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01084ce5-4229-4c46-bb4e-9c6973a000e4-bound-sa-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:37.627355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:37.627250    5074 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01084ce5-4229-4c46-bb4e-9c6973a000e4-ca-trust-extracted\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:38.343211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:38.343176    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6768769b75-2ms22" event={"ID":"01084ce5-4229-4c46-bb4e-9c6973a000e4","Type":"ContainerDied","Data":"efee1db92f8fddc6eebe1c2f3d9bfb3f028dc676b9d5bdf4d577fa3ba7664f2b"}
Nov 03 23:29:38.343211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:38.343199    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6768769b75-2ms22"
Nov 03 23:29:38.343672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:38.343245    5074 scope.go:122] "RemoveContainer" containerID="2ed59dec574492f59aff5b84c26161d9cc98ea261c6d4cc92e8d3953589309d2"
Nov 03 23:29:38.368001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:38.367977    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6768769b75-2ms22"]
Nov 03 23:29:38.372285 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:38.372259    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6768769b75-2ms22"]
Nov 03 23:29:39.016307 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:39.016276    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01084ce5-4229-4c46-bb4e-9c6973a000e4" path="/var/lib/kubelet/pods/01084ce5-4229-4c46-bb4e-9c6973a000e4/volumes"
Nov 03 23:29:44.380928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.380897    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-ddq79-rh2q9"]
Nov 03 23:29:44.381380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.381102    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-ddq79-rh2q9" podUID="4a23c360-8e0e-4985-ba27-a6631df95e11" containerName="insights-gathering" containerID="cri-o://2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af" gracePeriod=30
Nov 03 23:29:44.645934 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.645876    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:29:44.791942 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.791912    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") pod \"4a23c360-8e0e-4985-ba27-a6631df95e11\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") "
Nov 03 23:29:44.792109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.791951    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") pod \"4a23c360-8e0e-4985-ba27-a6631df95e11\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") "
Nov 03 23:29:44.792109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.792000    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") pod \"4a23c360-8e0e-4985-ba27-a6631df95e11\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") "
Nov 03 23:29:44.792109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.792064    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") pod \"4a23c360-8e0e-4985-ba27-a6631df95e11\" (UID: \"4a23c360-8e0e-4985-ba27-a6631df95e11\") "
Nov 03 23:29:44.792389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.792287    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle" pod "4a23c360-8e0e-4985-ba27-a6631df95e11" (UID: "4a23c360-8e0e-4985-ba27-a6631df95e11"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:44.792469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.792418    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path" pod "4a23c360-8e0e-4985-ba27-a6631df95e11" (UID: "4a23c360-8e0e-4985-ba27-a6631df95e11"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:29:44.792508 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.792463    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle" pod "4a23c360-8e0e-4985-ba27-a6631df95e11" (UID: "4a23c360-8e0e-4985-ba27-a6631df95e11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:44.794348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.794327    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz" pod "4a23c360-8e0e-4985-ba27-a6631df95e11" (UID: "4a23c360-8e0e-4985-ba27-a6631df95e11"). InnerVolumeSpecName "kube-api-access-vbhzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:44.893273 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.893248    5074 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4a23c360-8e0e-4985-ba27-a6631df95e11-archives-path\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:44.893273 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.893272    5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-trusted-ca-bundle\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:44.893413 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.893282    5074 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23c360-8e0e-4985-ba27-a6631df95e11-service-ca-bundle\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:44.893413 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:44.893291    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhzz\" (UniqueName: \"kubernetes.io/projected/4a23c360-8e0e-4985-ba27-a6631df95e11-kube-api-access-vbhzz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:45.376088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.376001    5074 generic.go:356] "Generic (PLEG): container finished" podID="4a23c360-8e0e-4985-ba27-a6631df95e11" containerID="2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af" exitCode=2
Nov 03 23:29:45.376088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.376074    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ddq79-rh2q9"
Nov 03 23:29:45.376295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.376093    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ddq79-rh2q9" event={"ID":"4a23c360-8e0e-4985-ba27-a6631df95e11","Type":"ContainerDied","Data":"2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af"}
Nov 03 23:29:45.376295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.376152    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ddq79-rh2q9" event={"ID":"4a23c360-8e0e-4985-ba27-a6631df95e11","Type":"ContainerDied","Data":"6d9b50efdd5bbed2f29bf7de9d13d454a607b89fee494995e709ace5a37181ee"}
Nov 03 23:29:45.376295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.376175    5074 scope.go:122] "RemoveContainer" containerID="2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af"
Nov 03 23:29:45.386069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.385843    5074 scope.go:122] "RemoveContainer" containerID="2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af"
Nov 03 23:29:45.386321 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:29:45.386133    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af\": container with ID starting with 2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af not found: ID does not exist" containerID="2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af"
Nov 03 23:29:45.386321 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.386161    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af"} err="failed to get container status \"2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af\": rpc error: code = NotFound desc = could not find container \"2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af\": container with ID starting with 2fde867f128f4b7040094a20bfc9331cf3f091bf67c6ece876c226c7f90479af not found: ID does not exist"
Nov 03 23:29:45.402189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.402158    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-ddq79-rh2q9"]
Nov 03 23:29:45.405874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:45.405853    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-ddq79-rh2q9"]
Nov 03 23:29:47.015723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:47.015690    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a23c360-8e0e-4985-ba27-a6631df95e11" path="/var/lib/kubelet/pods/4a23c360-8e0e-4985-ba27-a6631df95e11/volumes"
Nov 03 23:29:53.501925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.501882    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-55857b5d96-gc7z7" podUID="18e412ff-1d0b-4236-8686-401a852f8e3e" containerName="console" containerID="cri-o://9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de" gracePeriod=15
Nov 03 23:29:53.752418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.752358    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:53.870473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870442    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.870654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870498    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.870654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870530    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc\" (UniqueName: \"kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.870654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870559    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.870850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870815    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.870906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870853    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:29:53.870958 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870904    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.871015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870957    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:53.871015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.870968    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config\") pod \"18e412ff-1d0b-4236-8686-401a852f8e3e\" (UID: \"18e412ff-1d0b-4236-8686-401a852f8e3e\") "
Nov 03 23:29:53.871239 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.871138    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:53.871365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.871301    5074 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e412ff-1d0b-4236-8686-401a852f8e3e-tmp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:53.871365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.871317    5074 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-oauth-serving-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:53.871365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.871328    5074 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:53.871365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.871356    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:53.872948 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.872927    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "kube-api-access-zdrpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:53.873035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.872945    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:29:53.873035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.872959    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config" pod "18e412ff-1d0b-4236-8686-401a852f8e3e" (UID: "18e412ff-1d0b-4236-8686-401a852f8e3e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:29:53.972045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.972013    5074 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-serving-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:53.972045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.972040    5074 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18e412ff-1d0b-4236-8686-401a852f8e3e-console-config\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:53.972045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.972052    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrpc\" (UniqueName: \"kubernetes.io/projected/18e412ff-1d0b-4236-8686-401a852f8e3e-kube-api-access-zdrpc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:53.972270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:53.972062    5074 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18e412ff-1d0b-4236-8686-401a852f8e3e-console-oauth-config\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:54.419413 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.419379    5074 generic.go:356] "Generic (PLEG): container finished" podID="18e412ff-1d0b-4236-8686-401a852f8e3e" containerID="9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de" exitCode=0
Nov 03 23:29:54.419583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.419423    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55857b5d96-gc7z7"
Nov 03 23:29:54.419583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.419434    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55857b5d96-gc7z7" event={"ID":"18e412ff-1d0b-4236-8686-401a852f8e3e","Type":"ContainerDied","Data":"9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de"}
Nov 03 23:29:54.419583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.419473    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55857b5d96-gc7z7" event={"ID":"18e412ff-1d0b-4236-8686-401a852f8e3e","Type":"ContainerDied","Data":"4b47d257d358a08b404232424f7bf1313c7b9f860b7c8f7d234961a38d1bca77"}
Nov 03 23:29:54.419583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.419501    5074 scope.go:122] "RemoveContainer" containerID="9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de"
Nov 03 23:29:54.428029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.428011    5074 scope.go:122] "RemoveContainer" containerID="9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de"
Nov 03 23:29:54.428286 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:29:54.428265    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de\": container with ID starting with 9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de not found: ID does not exist" containerID="9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de"
Nov 03 23:29:54.428345 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.428298    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de"} err="failed to get container status \"9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de\": rpc error: code = NotFound desc = could not find container \"9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de\": container with ID starting with 9cb9e4e02aacb1df7e94a4e4fcf9563383f82c41f45f8278682830e29c09b5de not found: ID does not exist"
Nov 03 23:29:54.443981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.443961    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55857b5d96-gc7z7"]
Nov 03 23:29:54.445404 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:54.445383    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55857b5d96-gc7z7"]
Nov 03 23:29:55.014932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:55.014896    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e412ff-1d0b-4236-8686-401a852f8e3e" path="/var/lib/kubelet/pods/18e412ff-1d0b-4236-8686-401a852f8e3e/volumes"
Nov 03 23:29:58.020765 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:29:58.020707    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:29:59.326893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.326833    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95" podUID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" containerName="registry" containerID="cri-o://14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655" gracePeriod=30
Nov 03 23:29:59.580397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.580341    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:29:59.719803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.719767    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.719969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.719822    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.719969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.719854    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.719969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.719886    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.719969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.719912    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.720186 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.720008    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.720186 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.720065    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.720186 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.720109    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") pod \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\" (UID: \"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7\") "
Nov 03 23:29:59.720365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.720341    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:59.720442 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.720417    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:29:59.722258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.722225    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:59.722258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.722244    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:59.722626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.722606    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:29:59.722885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.722863    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "kube-api-access-425hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:29:59.722962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.722865    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:29:59.726915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.726895    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted" pod "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" (UID: "f74969b1-2e46-4e1c-ae7e-a56a2030b5b7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:29:59.821536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821506    5074 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-bound-sa-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821531    5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-trusted-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821541    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425hw\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-kube-api-access-425hw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821552    5074 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-certificates\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821562    5074 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-registry-tls\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821571    5074 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-installation-pull-secrets\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821580    5074 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-image-registry-private-configuration\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:29:59.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:29:59.821589    5074 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7-ca-trust-extracted\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.444829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.444794    5074 generic.go:356] "Generic (PLEG): container finished" podID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" containerID="14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655" exitCode=0
Nov 03 23:30:00.445315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.444851    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95"
Nov 03 23:30:00.445315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.444875    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95" event={"ID":"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7","Type":"ContainerDied","Data":"14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655"}
Nov 03 23:30:00.445315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.444929    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b96cb8d6-ctr95" event={"ID":"f74969b1-2e46-4e1c-ae7e-a56a2030b5b7","Type":"ContainerDied","Data":"f68e05fb4588f7170ca9eac33ba7c9eb52fcbef0e3b14cc89c1e395aec111472"}
Nov 03 23:30:00.445315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.444949    5074 scope.go:122] "RemoveContainer" containerID="14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655"
Nov 03 23:30:00.453993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.453977    5074 scope.go:122] "RemoveContainer" containerID="14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655"
Nov 03 23:30:00.454246 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:30:00.454228    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655\": container with ID starting with 14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655 not found: ID does not exist" containerID="14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655"
Nov 03 23:30:00.454300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.454258    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655"} err="failed to get container status \"14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655\": rpc error: code = NotFound desc = could not find container \"14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655\": container with ID starting with 14005c75bd4000b586ae95c1881de36561d597f340751881d575c91b549f5655 not found: ID does not exist"
Nov 03 23:30:00.469383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.469361    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b96cb8d6-ctr95"]
Nov 03 23:30:00.474253 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:00.474224    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b96cb8d6-ctr95"]
Nov 03 23:30:01.015204 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:01.015172    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" path="/var/lib/kubelet/pods/f74969b1-2e46-4e1c-ae7e-a56a2030b5b7/volumes"
Nov 03 23:30:17.516926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:17.516880    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nsh2h" event={"ID":"001ac09b-53c8-4437-bc59-d1785cc36723","Type":"ContainerStarted","Data":"7e63b6642fac79a26ef4ace562d8fee177a7da23abbf2e92eb92229e009c9509"}
Nov 03 23:30:17.517448 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:17.517160    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:30:17.539671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:17.539634    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nsh2h"
Nov 03 23:30:17.543419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:17.543351    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-nsh2h" podStartSLOduration=1.523811864 podStartE2EDuration="1m5.543336343s" totalImagesPullingTime="1m4.019524479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:30:17.542263416 +0000 UTC m=+82.641863561" watchObservedRunningTime="2026-11-03 23:30:17.543336343 +0000 UTC m=+82.642936485"
Nov 03 23:30:28.020345 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:30:28.020303    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:30:38.604969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:38.604930    5074 generic.go:356] "Generic (PLEG): container finished" podID="96b959e5-0c65-457f-8f9b-d386854c3642" containerID="0134582ffee02c4be29da39bb8c2c00c389a70b644f574a87f829ed967feb3f2" exitCode=0
Nov 03 23:30:38.605373 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:38.605003    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz" event={"ID":"96b959e5-0c65-457f-8f9b-d386854c3642","Type":"ContainerDied","Data":"0134582ffee02c4be29da39bb8c2c00c389a70b644f574a87f829ed967feb3f2"}
Nov 03 23:30:39.758930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.758904    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:30:39.841968 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.841936    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle\") pod \"96b959e5-0c65-457f-8f9b-d386854c3642\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") "
Nov 03 23:30:39.842130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.841995    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path\") pod \"96b959e5-0c65-457f-8f9b-d386854c3642\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") "
Nov 03 23:30:39.842130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.842045    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle\") pod \"96b959e5-0c65-457f-8f9b-d386854c3642\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") "
Nov 03 23:30:39.842130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.842089    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6\" (UniqueName: \"kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6\") pod \"96b959e5-0c65-457f-8f9b-d386854c3642\" (UID: \"96b959e5-0c65-457f-8f9b-d386854c3642\") "
Nov 03 23:30:39.842384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.842361    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle" pod "96b959e5-0c65-457f-8f9b-d386854c3642" (UID: "96b959e5-0c65-457f-8f9b-d386854c3642"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:30:39.842452 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.842386    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path" pod "96b959e5-0c65-457f-8f9b-d386854c3642" (UID: "96b959e5-0c65-457f-8f9b-d386854c3642"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:30:39.842507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.842478    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle" pod "96b959e5-0c65-457f-8f9b-d386854c3642" (UID: "96b959e5-0c65-457f-8f9b-d386854c3642"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:30:39.844202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.844179    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6" pod "96b959e5-0c65-457f-8f9b-d386854c3642" (UID: "96b959e5-0c65-457f-8f9b-d386854c3642"). InnerVolumeSpecName "kube-api-access-cxth6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:30:39.942590 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.942565    5074 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-service-ca-bundle\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:39.942590 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.942591    5074 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b959e5-0c65-457f-8f9b-d386854c3642-archives-path\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:39.942784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.942602    5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b959e5-0c65-457f-8f9b-d386854c3642-trusted-ca-bundle\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:39.942784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:39.942612    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxth6\" (UniqueName: \"kubernetes.io/projected/96b959e5-0c65-457f-8f9b-d386854c3642-kube-api-access-cxth6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:40.613733 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:40.613701    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz"
Nov 03 23:30:40.613733 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:40.613709    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w5bsk-9tpvz" event={"ID":"96b959e5-0c65-457f-8f9b-d386854c3642","Type":"ContainerDied","Data":"b9a0f50267f9d509d8de224f86978079214206f212d631a2e5a10149c43256f4"}
Nov 03 23:30:40.613957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:30:40.613791    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9a0f50267f9d509d8de224f86978079214206f212d631a2e5a10149c43256f4"
Nov 03 23:30:58.020484 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:30:58.020437    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:31:28.020471 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:31:28.020391    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:31:58.019042 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:31:58.018998    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:32:28.020179 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:32:28.020135    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:32:31.564595 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564561    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"]
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564873    5074 cpu_manager.go:414] "containerMap: removing container" podUID="96b959e5-0c65-457f-8f9b-d386854c3642" containerName="insights-gathering"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564884    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b959e5-0c65-457f-8f9b-d386854c3642" containerName="insights-gathering"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564902    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4a23c360-8e0e-4985-ba27-a6631df95e11" containerName="insights-gathering"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564907    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a23c360-8e0e-4985-ba27-a6631df95e11" containerName="insights-gathering"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564929    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" containerName="registry"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564934    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" containerName="registry"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564944    5074 cpu_manager.go:414] "containerMap: removing container" podUID="01084ce5-4229-4c46-bb4e-9c6973a000e4" containerName="registry"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564948    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01084ce5-4229-4c46-bb4e-9c6973a000e4" containerName="registry"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564955    5074 cpu_manager.go:414] "containerMap: removing container" podUID="18e412ff-1d0b-4236-8686-401a852f8e3e" containerName="console"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.564960    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e412ff-1d0b-4236-8686-401a852f8e3e" containerName="console"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.565018    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74969b1-2e46-4e1c-ae7e-a56a2030b5b7" containerName="registry"
Nov 03 23:32:31.565021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.565025    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="01084ce5-4229-4c46-bb4e-9c6973a000e4" containerName="registry"
Nov 03 23:32:31.565420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.565031    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e412ff-1d0b-4236-8686-401a852f8e3e" containerName="console"
Nov 03 23:32:31.565420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.565037    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b959e5-0c65-457f-8f9b-d386854c3642" containerName="insights-gathering"
Nov 03 23:32:31.565420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.565045    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a23c360-8e0e-4985-ba27-a6631df95e11" containerName="insights-gathering"
Nov 03 23:32:31.567097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.567080    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.573569 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.573539    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 23:32:31.573699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.573645    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 23:32:31.578079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.578057    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"]
Nov 03 23:32:31.645258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.645234    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6j4q7\""
Nov 03 23:32:31.692694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.692660    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"]
Nov 03 23:32:31.694971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.694950    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.700476 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.700457    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sh6t7\""
Nov 03 23:32:31.704757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.704721    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfp5\" (UniqueName: \"kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.704853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.704818    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.704906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.704892    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.709861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.709842    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 23:32:31.712282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.712255    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"]
Nov 03 23:32:31.799268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.799243    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"]
Nov 03 23:32:31.801319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.801304    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.805426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805394    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.805562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805435    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332db91-1798-471f-87d8-55217be24115-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.805562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805468    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.805678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805617    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfp5\" (UniqueName: \"kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.805678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805648    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.805806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805684    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.805806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805715    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332db91-1798-471f-87d8-55217be24115-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.805806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805755    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332db91-1798-471f-87d8-55217be24115-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.805806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805790    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt25d\" (UniqueName: \"kubernetes.io/projected/c332db91-1798-471f-87d8-55217be24115-kube-api-access-lt25d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.806018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805825    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.806018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805854    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.806018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805903    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.806018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.805909    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.806212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.806183    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.811554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.811536    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-896wj\""
Nov 03 23:32:31.820301 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.820242    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"]
Nov 03 23:32:31.834372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.834352    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfp5\" (UniqueName: \"kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.881082 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.881051    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:31.906435 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906399    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.906543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906460    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332db91-1798-471f-87d8-55217be24115-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.906543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906510    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f702aa70-1ecd-4e89-a575-04034b778897-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.906697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906677    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.906780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906723    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.906869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906850    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f702aa70-1ecd-4e89-a575-04034b778897-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.906869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906854    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.906963 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906893    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.906973    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332db91-1798-471f-87d8-55217be24115-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907004    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332db91-1798-471f-87d8-55217be24115-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907045    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt25d\" (UniqueName: \"kubernetes.io/projected/c332db91-1798-471f-87d8-55217be24115-kube-api-access-lt25d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907081    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907110    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.907164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907143    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glq5g\" (UniqueName: \"kubernetes.io/projected/f702aa70-1ecd-4e89-a575-04034b778897-kube-api-access-glq5g\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.907318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907177    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907212    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.907318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907230    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907248    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.907597 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907575    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907626    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f702aa70-1ecd-4e89-a575-04034b778897-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:31.907677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907637    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.907804 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907713    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332db91-1798-471f-87d8-55217be24115-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.908004 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.907977    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.909398 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.909375    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332db91-1798-471f-87d8-55217be24115-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.909584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.909563    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332db91-1798-471f-87d8-55217be24115-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.928831 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.924057    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt25d\" (UniqueName: \"kubernetes.io/projected/c332db91-1798-471f-87d8-55217be24115-kube-api-access-lt25d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:31.929619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:31.929595    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332db91-1798-471f-87d8-55217be24115-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kchxb\" (UID: \"c332db91-1798-471f-87d8-55217be24115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:32.006094 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.006057    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:32.008039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008011    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f702aa70-1ecd-4e89-a575-04034b778897-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008051    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008069    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008096    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f702aa70-1ecd-4e89-a575-04034b778897-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008134    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008156    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glq5g\" (UniqueName: \"kubernetes.io/projected/f702aa70-1ecd-4e89-a575-04034b778897-kube-api-access-glq5g\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008192    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008220    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008259    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f702aa70-1ecd-4e89-a575-04034b778897-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008486    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008881    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.008972 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.008911    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f702aa70-1ecd-4e89-a575-04034b778897-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.009178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.009155    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.009274 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.009237    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.010784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.010734    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f702aa70-1ecd-4e89-a575-04034b778897-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.011114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.011092    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f702aa70-1ecd-4e89-a575-04034b778897-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.022440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.022416    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f702aa70-1ecd-4e89-a575-04034b778897-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.022520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.022485    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glq5g\" (UniqueName: \"kubernetes.io/projected/f702aa70-1ecd-4e89-a575-04034b778897-kube-api-access-glq5g\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk\" (UID: \"f702aa70-1ecd-4e89-a575-04034b778897\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.114425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.114355    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:32.163624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.163535    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"]
Nov 03 23:32:32.172808 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:32:32.172731    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3e2a17_1ed5_4746_91f1_b9423f2af05b.slice/crio-75776e8d0ebc7fe15ef73338b5873e0cdad54fc922f2590d769923becd083eb6 WatchSource:0}: Error finding container 75776e8d0ebc7fe15ef73338b5873e0cdad54fc922f2590d769923becd083eb6: Status 404 returned error can't find the container with id 75776e8d0ebc7fe15ef73338b5873e0cdad54fc922f2590d769923becd083eb6
Nov 03 23:32:32.319510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.319475    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"]
Nov 03 23:32:32.323679 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:32:32.323648    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc332db91_1798_471f_87d8_55217be24115.slice/crio-0a74f99442af74bd52a3f698b083b29f162ca9e8a9e5f7328903c1d564f11789 WatchSource:0}: Error finding container 0a74f99442af74bd52a3f698b083b29f162ca9e8a9e5f7328903c1d564f11789: Status 404 returned error can't find the container with id 0a74f99442af74bd52a3f698b083b29f162ca9e8a9e5f7328903c1d564f11789
Nov 03 23:32:32.391451 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:32.391388    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"]
Nov 03 23:32:32.394357 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:32:32.394332    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf702aa70_1ecd_4e89_a575_04034b778897.slice/crio-bc1847f979d9cec77113a9127439181ac70e5c7cf28cb1a01f84dee1251b8b4d WatchSource:0}: Error finding container bc1847f979d9cec77113a9127439181ac70e5c7cf28cb1a01f84dee1251b8b4d: Status 404 returned error can't find the container with id bc1847f979d9cec77113a9127439181ac70e5c7cf28cb1a01f84dee1251b8b4d
Nov 03 23:32:33.058903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:33.058862    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95" event={"ID":"4b3e2a17-1ed5-4746-91f1-b9423f2af05b","Type":"ContainerStarted","Data":"75776e8d0ebc7fe15ef73338b5873e0cdad54fc922f2590d769923becd083eb6"}
Nov 03 23:32:33.060198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:33.060174    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk" event={"ID":"f702aa70-1ecd-4e89-a575-04034b778897","Type":"ContainerStarted","Data":"bc1847f979d9cec77113a9127439181ac70e5c7cf28cb1a01f84dee1251b8b4d"}
Nov 03 23:32:33.061484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:33.061452    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" event={"ID":"c332db91-1798-471f-87d8-55217be24115","Type":"ContainerStarted","Data":"0a74f99442af74bd52a3f698b083b29f162ca9e8a9e5f7328903c1d564f11789"}
Nov 03 23:32:36.064230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:36.064196    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:32:36.064735 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:36.064284    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:32:36.064735 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:36.064333    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:32:36.368932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:36.368897    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:32:36.369066 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:36.368964    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:32:36.369066 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:36.368996    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:32:37.086329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:37.086292    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk" event={"ID":"f702aa70-1ecd-4e89-a575-04034b778897","Type":"ContainerStarted","Data":"f84ead90dc73d02d283457dbf9828b774f54c74bea8eb1d12504c9514e22ea6f"}
Nov 03 23:32:37.088310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:37.088274    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" event={"ID":"c332db91-1798-471f-87d8-55217be24115","Type":"ContainerStarted","Data":"6a55d18e08d63d1cf9178955e9b9aca3694ccec6700e6e44cfd0ca08e5c9cb24"}
Nov 03 23:32:37.114118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:37.114070    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk" podStartSLOduration=2.142198362 podStartE2EDuration="6.114057024s" totalImagesPullingTime="3.971858662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:37.112705329 +0000 UTC m=+222.212305478" watchObservedRunningTime="2026-11-03 23:32:37.114057024 +0000 UTC m=+222.213657168"
Nov 03 23:32:37.115375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:37.115352    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:37.120485 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:37.120464    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:37.141471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:37.141422    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podStartSLOduration=2.403689013 podStartE2EDuration="6.141408716s" totalImagesPullingTime="3.737719703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:37.14049187 +0000 UTC m=+222.240092018" watchObservedRunningTime="2026-11-03 23:32:37.141408716 +0000 UTC m=+222.241008926"
Nov 03 23:32:38.007324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:38.007286    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:38.013024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:38.012998    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:38.093020 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:38.092976    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:38.093417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:38.093039    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:38.094003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:38.093972    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:32:38.094099 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:38.094039    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk"
Nov 03 23:32:39.097583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:39.097543    5074 generic.go:356] "Generic (PLEG): container finished" podID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerID="74d5c4052b785c209f1e4f558dbb90731723855694c3360f5867f7e707c377c6" exitCode=0
Nov 03 23:32:39.098000 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:39.097626    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95" event={"ID":"4b3e2a17-1ed5-4746-91f1-b9423f2af05b","Type":"ContainerDied","Data":"74d5c4052b785c209f1e4f558dbb90731723855694c3360f5867f7e707c377c6"}
Nov 03 23:32:42.111347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:42.111312    5074 generic.go:356] "Generic (PLEG): container finished" podID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerID="7c2e87e77fa01bcfa98dc7ac4eb5700966aaca3168bdf472b2cdc0082b015f69" exitCode=0
Nov 03 23:32:42.111784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:42.111395    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95" event={"ID":"4b3e2a17-1ed5-4746-91f1-b9423f2af05b","Type":"ContainerDied","Data":"7c2e87e77fa01bcfa98dc7ac4eb5700966aaca3168bdf472b2cdc0082b015f69"}
Nov 03 23:32:48.141425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:48.141340    5074 generic.go:356] "Generic (PLEG): container finished" podID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerID="be8b7b1af2e89d7324b3ec693a6633b9432d1bb852ca7829528644f0441c7784" exitCode=0
Nov 03 23:32:48.141425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:48.141412    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95" event={"ID":"4b3e2a17-1ed5-4746-91f1-b9423f2af05b","Type":"ContainerDied","Data":"be8b7b1af2e89d7324b3ec693a6633b9432d1bb852ca7829528644f0441c7784"}
Nov 03 23:32:49.271810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.271789    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:49.469488 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.469463    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util\") pod \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") "
Nov 03 23:32:49.469644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.469518    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5\" (UniqueName: \"kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5\") pod \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") "
Nov 03 23:32:49.469644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.469563    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle\") pod \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\" (UID: \"4b3e2a17-1ed5-4746-91f1-b9423f2af05b\") "
Nov 03 23:32:49.470414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.470380    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle" pod "4b3e2a17-1ed5-4746-91f1-b9423f2af05b" (UID: "4b3e2a17-1ed5-4746-91f1-b9423f2af05b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:32:49.471790 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.471766    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5" pod "4b3e2a17-1ed5-4746-91f1-b9423f2af05b" (UID: "4b3e2a17-1ed5-4746-91f1-b9423f2af05b"). InnerVolumeSpecName "kube-api-access-4qfp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:32:49.474017 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.473997    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util" pod "4b3e2a17-1ed5-4746-91f1-b9423f2af05b" (UID: "4b3e2a17-1ed5-4746-91f1-b9423f2af05b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:32:49.570692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.570669    5074 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-util\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:32:49.570692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.570691    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qfp5\" (UniqueName: \"kubernetes.io/projected/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-kube-api-access-4qfp5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:32:49.570854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:49.570702    5074 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3e2a17-1ed5-4746-91f1-b9423f2af05b-bundle\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:32:50.151961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:50.151929    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95" event={"ID":"4b3e2a17-1ed5-4746-91f1-b9423f2af05b","Type":"ContainerDied","Data":"75776e8d0ebc7fe15ef73338b5873e0cdad54fc922f2590d769923becd083eb6"}
Nov 03 23:32:50.152129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:50.151977    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75776e8d0ebc7fe15ef73338b5873e0cdad54fc922f2590d769923becd083eb6"
Nov 03 23:32:50.152129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:50.151986    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cknz95"
Nov 03 23:32:55.594410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594378    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"]
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594689    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="extract"
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594700    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="extract"
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594731    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="util"
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594749    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="util"
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594756    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="pull"
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594761    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="pull"
Nov 03 23:32:55.594815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.594816    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3e2a17-1ed5-4746-91f1-b9423f2af05b" containerName="extract"
Nov 03 23:32:55.643502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.643476    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"]
Nov 03 23:32:55.643641 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.643572    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.652459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.652432    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:32:55.652578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.652476    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-79t2c\""
Nov 03 23:32:55.652578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.652503    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 23:32:55.712004 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.711981    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a63f2c-0cd4-4c20-915b-46f516b0bb4c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-trzg9\" (UID: \"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.712114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.712029    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshg2\" (UniqueName: \"kubernetes.io/projected/a8a63f2c-0cd4-4c20-915b-46f516b0bb4c-kube-api-access-kshg2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-trzg9\" (UID: \"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.812459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.812424    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a63f2c-0cd4-4c20-915b-46f516b0bb4c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-trzg9\" (UID: \"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.812604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.812491    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshg2\" (UniqueName: \"kubernetes.io/projected/a8a63f2c-0cd4-4c20-915b-46f516b0bb4c-kube-api-access-kshg2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-trzg9\" (UID: \"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.812809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.812789    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a63f2c-0cd4-4c20-915b-46f516b0bb4c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-trzg9\" (UID: \"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.838455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.838428    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshg2\" (UniqueName: \"kubernetes.io/projected/a8a63f2c-0cd4-4c20-915b-46f516b0bb4c-kube-api-access-kshg2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-trzg9\" (UID: \"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:55.965483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:55.965455    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"
Nov 03 23:32:56.255796 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:56.255767    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9"]
Nov 03 23:32:56.258420 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:32:56.258388    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a63f2c_0cd4_4c20_915b_46f516b0bb4c.slice/crio-5ec7745da48b556d360b75542aa0eebbfada2ee9c89dad440862c1b370e0ca1b WatchSource:0}: Error finding container 5ec7745da48b556d360b75542aa0eebbfada2ee9c89dad440862c1b370e0ca1b: Status 404 returned error can't find the container with id 5ec7745da48b556d360b75542aa0eebbfada2ee9c89dad440862c1b370e0ca1b
Nov 03 23:32:57.180958 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:32:57.180922    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9" event={"ID":"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c","Type":"ContainerStarted","Data":"5ec7745da48b556d360b75542aa0eebbfada2ee9c89dad440862c1b370e0ca1b"}
Nov 03 23:32:58.020845 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:32:58.020806    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:33:01.201779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:01.201732    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9" event={"ID":"a8a63f2c-0cd4-4c20-915b-46f516b0bb4c","Type":"ContainerStarted","Data":"fe7cfa00f122c6259d96ec1e00d4ff0b953adffa3d5a2f7c24224db4dffacc6e"}
Nov 03 23:33:01.244134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:01.244058    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-trzg9" podStartSLOduration=1.657468742 podStartE2EDuration="6.244044183s" totalImagesPullingTime="4.586575441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:01.243626811 +0000 UTC m=+246.343226955" watchObservedRunningTime="2026-11-03 23:33:01.244044183 +0000 UTC m=+246.343644326"
Nov 03 23:33:09.026684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.026649    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"]
Nov 03 23:33:09.030158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.030141    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.037386 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.037362    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-m64nv\""
Nov 03 23:33:09.037386 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.037364    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 23:33:09.037531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.037416    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 23:33:09.039806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.039782    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"]
Nov 03 23:33:09.123554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.123511    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fkkf\" (UniqueName: \"kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.123788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.123661    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.123788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.123692    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.224730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.224686    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.224730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.224730    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.224999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.224791    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fkkf\" (UniqueName: \"kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.225085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.225065    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.225139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.225098    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.238606 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.238581    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fkkf\" (UniqueName: \"kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.348445 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.348371    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:09.622456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:09.622359    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"]
Nov 03 23:33:09.625263 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:33:09.625231    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a641be_db2c_42a4_a41c_85fe74036c76.slice/crio-77db5d4ecb49840d8f0dec97d7388f1a8df1ef0b638e94f253d0ea5f5f8f3b3e WatchSource:0}: Error finding container 77db5d4ecb49840d8f0dec97d7388f1a8df1ef0b638e94f253d0ea5f5f8f3b3e: Status 404 returned error can't find the container with id 77db5d4ecb49840d8f0dec97d7388f1a8df1ef0b638e94f253d0ea5f5f8f3b3e
Nov 03 23:33:10.241288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:10.241251    5074 generic.go:356] "Generic (PLEG): container finished" podID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerID="de1b5e5bc3b7d17e5eba58ef6e28c66b2399ae099c7f35d9e6cf8b52beab5b90" exitCode=0
Nov 03 23:33:10.241719 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:10.241337    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m" event={"ID":"e7a641be-db2c-42a4-a41c-85fe74036c76","Type":"ContainerDied","Data":"de1b5e5bc3b7d17e5eba58ef6e28c66b2399ae099c7f35d9e6cf8b52beab5b90"}
Nov 03 23:33:10.241719 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:10.241384    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m" event={"ID":"e7a641be-db2c-42a4-a41c-85fe74036c76","Type":"ContainerStarted","Data":"77db5d4ecb49840d8f0dec97d7388f1a8df1ef0b638e94f253d0ea5f5f8f3b3e"}
Nov 03 23:33:11.247718 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:11.247681    5074 generic.go:356] "Generic (PLEG): container finished" podID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerID="874c60415eae03a7c3715c81b72e732723dff42f0978f7053538f5db6db81bd9" exitCode=0
Nov 03 23:33:11.248258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:11.247766    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m" event={"ID":"e7a641be-db2c-42a4-a41c-85fe74036c76","Type":"ContainerDied","Data":"874c60415eae03a7c3715c81b72e732723dff42f0978f7053538f5db6db81bd9"}
Nov 03 23:33:12.254761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:12.254699    5074 generic.go:356] "Generic (PLEG): container finished" podID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerID="e08ea8e76907a390a3de9bf1a64351183b9249bd27a678bbc6bd8980e1c97af5" exitCode=0
Nov 03 23:33:12.255157 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:12.254776    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m" event={"ID":"e7a641be-db2c-42a4-a41c-85fe74036c76","Type":"ContainerDied","Data":"e08ea8e76907a390a3de9bf1a64351183b9249bd27a678bbc6bd8980e1c97af5"}
Nov 03 23:33:13.387500 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.387476    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:13.461376 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.461345    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util\") pod \"e7a641be-db2c-42a4-a41c-85fe74036c76\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") "
Nov 03 23:33:13.461573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.461384    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf\" (UniqueName: \"kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf\") pod \"e7a641be-db2c-42a4-a41c-85fe74036c76\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") "
Nov 03 23:33:13.461573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.461425    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle\") pod \"e7a641be-db2c-42a4-a41c-85fe74036c76\" (UID: \"e7a641be-db2c-42a4-a41c-85fe74036c76\") "
Nov 03 23:33:13.462074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.462041    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle" pod "e7a641be-db2c-42a4-a41c-85fe74036c76" (UID: "e7a641be-db2c-42a4-a41c-85fe74036c76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:33:13.463587 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.463563    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf" pod "e7a641be-db2c-42a4-a41c-85fe74036c76" (UID: "e7a641be-db2c-42a4-a41c-85fe74036c76"). InnerVolumeSpecName "kube-api-access-8fkkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:33:13.466560 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.466535    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util" pod "e7a641be-db2c-42a4-a41c-85fe74036c76" (UID: "e7a641be-db2c-42a4-a41c-85fe74036c76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:33:13.562409 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.562333    5074 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-util\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:33:13.562409 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.562363    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fkkf\" (UniqueName: \"kubernetes.io/projected/e7a641be-db2c-42a4-a41c-85fe74036c76-kube-api-access-8fkkf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:33:13.562409 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:13.562375    5074 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7a641be-db2c-42a4-a41c-85fe74036c76-bundle\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:33:14.265707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:14.265672    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m" event={"ID":"e7a641be-db2c-42a4-a41c-85fe74036c76","Type":"ContainerDied","Data":"77db5d4ecb49840d8f0dec97d7388f1a8df1ef0b638e94f253d0ea5f5f8f3b3e"}
Nov 03 23:33:14.265707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:14.265696    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dcd7m"
Nov 03 23:33:14.265929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:14.265724    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77db5d4ecb49840d8f0dec97d7388f1a8df1ef0b638e94f253d0ea5f5f8f3b3e"
Nov 03 23:33:22.745302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745272    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"]
Nov 03 23:33:22.745716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745621    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="pull"
Nov 03 23:33:22.745716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745633    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="pull"
Nov 03 23:33:22.745716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745648    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="util"
Nov 03 23:33:22.745716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745653    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="util"
Nov 03 23:33:22.745716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745670    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="extract"
Nov 03 23:33:22.745716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745675    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="extract"
Nov 03 23:33:22.745943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.745750    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a641be-db2c-42a4-a41c-85fe74036c76" containerName="extract"
Nov 03 23:33:22.749173 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.749157    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.771860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.771836    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 23:33:22.772113 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.772096    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 23:33:22.772597 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.772583    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 23:33:22.772851 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.772839    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 23:33:22.773156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.773140    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 23:33:22.773221 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.773206    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bc7w7\""
Nov 03 23:33:22.784824 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.784801    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"]
Nov 03 23:33:22.839763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.839715    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.839893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.839776    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7148dc-308e-4a93-8f1d-1f8c0172c327-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.839893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.839847    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d7148dc-308e-4a93-8f1d-1f8c0172c327-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.839983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.839912    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrfj\" (UniqueName: \"kubernetes.io/projected/8d7148dc-308e-4a93-8f1d-1f8c0172c327-kube-api-access-lkrfj\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.839983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.839940    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.840122 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.840061    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.940931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.940898    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.941053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.940937    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7148dc-308e-4a93-8f1d-1f8c0172c327-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.941053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.940952    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d7148dc-308e-4a93-8f1d-1f8c0172c327-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.941053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.940985    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrfj\" (UniqueName: \"kubernetes.io/projected/8d7148dc-308e-4a93-8f1d-1f8c0172c327-kube-api-access-lkrfj\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.941053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.941009    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.941053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.941040    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.941932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.941898    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d7148dc-308e-4a93-8f1d-1f8c0172c327-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.947516 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.947485    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7148dc-308e-4a93-8f1d-1f8c0172c327-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.949496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.949470    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.949613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.949475    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.949613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.949475    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d7148dc-308e-4a93-8f1d-1f8c0172c327-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:22.954632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:22.954610    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrfj\" (UniqueName: \"kubernetes.io/projected/8d7148dc-308e-4a93-8f1d-1f8c0172c327-kube-api-access-lkrfj\") pod \"opendatahub-operator-controller-manager-775bfbdf99-tbczr\" (UID: \"8d7148dc-308e-4a93-8f1d-1f8c0172c327\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:23.063448 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.063367    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:23.407425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.407395    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"]
Nov 03 23:33:23.407837 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:33:23.407816    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7148dc_308e_4a93_8f1d_1f8c0172c327.slice/crio-740e2bb933325a3e760cda9708c238e08b78a495df420352e5410a817c4dae9f WatchSource:0}: Error finding container 740e2bb933325a3e760cda9708c238e08b78a495df420352e5410a817c4dae9f: Status 404 returned error can't find the container with id 740e2bb933325a3e760cda9708c238e08b78a495df420352e5410a817c4dae9f
Nov 03 23:33:23.607338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.607308    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"]
Nov 03 23:33:23.611801 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.611783    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:23.621035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.621014    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 23:33:23.621035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.621030    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 23:33:23.621975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.621952    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-k28n8\""
Nov 03 23:33:23.623411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.623392    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"]
Nov 03 23:33:23.747178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.747145    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rps5\" (UniqueName: \"kubernetes.io/projected/9c62bf5c-48e7-4250-ba5d-5e1a29efbe26-kube-api-access-9rps5\") pod \"authorino-operator-7b7bb4bc77-qnvzl\" (UID: \"9c62bf5c-48e7-4250-ba5d-5e1a29efbe26\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:23.847981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.847954    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rps5\" (UniqueName: \"kubernetes.io/projected/9c62bf5c-48e7-4250-ba5d-5e1a29efbe26-kube-api-access-9rps5\") pod \"authorino-operator-7b7bb4bc77-qnvzl\" (UID: \"9c62bf5c-48e7-4250-ba5d-5e1a29efbe26\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:23.861381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.861358    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rps5\" (UniqueName: \"kubernetes.io/projected/9c62bf5c-48e7-4250-ba5d-5e1a29efbe26-kube-api-access-9rps5\") pod \"authorino-operator-7b7bb4bc77-qnvzl\" (UID: \"9c62bf5c-48e7-4250-ba5d-5e1a29efbe26\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:23.926431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:23.926406    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:24.233963 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:24.233935    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"]
Nov 03 23:33:24.236383 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:33:24.236350    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c62bf5c_48e7_4250_ba5d_5e1a29efbe26.slice/crio-76899470d61802283d62dfcfba2b440e1f731ad6bccfebb706abd60ca44f6625 WatchSource:0}: Error finding container 76899470d61802283d62dfcfba2b440e1f731ad6bccfebb706abd60ca44f6625: Status 404 returned error can't find the container with id 76899470d61802283d62dfcfba2b440e1f731ad6bccfebb706abd60ca44f6625
Nov 03 23:33:24.311238 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:24.311201    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl" event={"ID":"9c62bf5c-48e7-4250-ba5d-5e1a29efbe26","Type":"ContainerStarted","Data":"76899470d61802283d62dfcfba2b440e1f731ad6bccfebb706abd60ca44f6625"}
Nov 03 23:33:24.312956 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:24.312928    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr" event={"ID":"8d7148dc-308e-4a93-8f1d-1f8c0172c327","Type":"ContainerStarted","Data":"740e2bb933325a3e760cda9708c238e08b78a495df420352e5410a817c4dae9f"}
Nov 03 23:33:27.330372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:27.330327    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr" event={"ID":"8d7148dc-308e-4a93-8f1d-1f8c0172c327","Type":"ContainerStarted","Data":"35c7023a70134858dad8bf1f8836bbfa67741dfd798a94527003f07923fb85a7"}
Nov 03 23:33:27.331891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:27.331865    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl" event={"ID":"9c62bf5c-48e7-4250-ba5d-5e1a29efbe26","Type":"ContainerStarted","Data":"188625bc78035d36457d4a8c4f9e5a2d2a3fbd90118dab6bae7fc27e7f8da7e8"}
Nov 03 23:33:27.332024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:27.332007    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:27.431584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:27.431485    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl" podStartSLOduration=1.609023146 podStartE2EDuration="4.431471216s" totalImagesPullingTime="2.82244807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:27.430249765 +0000 UTC m=+272.529849909" watchObservedRunningTime="2026-11-03 23:33:27.431471216 +0000 UTC m=+272.531071359"
Nov 03 23:33:28.020641 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:33:28.020607    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:33:28.337907 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:28.337822    5074 generic.go:356] "Generic (PLEG): container finished" podID="8d7148dc-308e-4a93-8f1d-1f8c0172c327" containerID="35c7023a70134858dad8bf1f8836bbfa67741dfd798a94527003f07923fb85a7" exitCode=0
Nov 03 23:33:28.338404 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:28.337918    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr" event={"ID":"8d7148dc-308e-4a93-8f1d-1f8c0172c327","Type":"ContainerDied","Data":"35c7023a70134858dad8bf1f8836bbfa67741dfd798a94527003f07923fb85a7"}
Nov 03 23:33:29.344165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:29.344128    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr" event={"ID":"8d7148dc-308e-4a93-8f1d-1f8c0172c327","Type":"ContainerStarted","Data":"9bdbd263b973239e77b2b45dde711d38bdc22e19e20029731c5527eef48b1f73"}
Nov 03 23:33:29.344550 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:29.344223    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:29.406941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:29.406870    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr" podStartSLOduration=2.958981677 podStartE2EDuration="7.40685666s" totalImagesPullingTime="3.86904096s" totalInitContainerRuntime="578.834023ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:29.404766621 +0000 UTC m=+274.504366762" watchObservedRunningTime="2026-11-03 23:33:29.40685666 +0000 UTC m=+274.506456804"
Nov 03 23:33:31.156402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.156367    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"]
Nov 03 23:33:31.159848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.159832    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.166845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.166826    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n88b2\""
Nov 03 23:33:31.182108 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.182081    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"]
Nov 03 23:33:31.319043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.319006    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w2cn8\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.319215 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.319062    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbg9\" (UniqueName: \"kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9\") pod \"kuadrant-operator-controller-manager-678d87766f-w2cn8\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.419802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.419689    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w2cn8\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.419802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.419791    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbg9\" (UniqueName: \"kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9\") pod \"kuadrant-operator-controller-manager-678d87766f-w2cn8\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.420061 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.420039    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w2cn8\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.450683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.450651    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbg9\" (UniqueName: \"kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9\") pod \"kuadrant-operator-controller-manager-678d87766f-w2cn8\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.471749 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.471721    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:31.764857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:31.764830    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"]
Nov 03 23:33:31.767313 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:33:31.767281    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5dacd3_94a5_4e52_bdb4_26f2fe871a14.slice/crio-188a1e80039604add7a422bafadc6261b7547b6fdeb09111f9a52696d95a24f0 WatchSource:0}: Error finding container 188a1e80039604add7a422bafadc6261b7547b6fdeb09111f9a52696d95a24f0: Status 404 returned error can't find the container with id 188a1e80039604add7a422bafadc6261b7547b6fdeb09111f9a52696d95a24f0
Nov 03 23:33:32.358853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:32.358814    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8" event={"ID":"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14","Type":"ContainerStarted","Data":"188a1e80039604add7a422bafadc6261b7547b6fdeb09111f9a52696d95a24f0"}
Nov 03 23:33:37.384258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:37.384224    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8" event={"ID":"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14","Type":"ContainerStarted","Data":"f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82"}
Nov 03 23:33:37.384593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:37.384353    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:37.415918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:37.415852    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8" podStartSLOduration=0.905803357 podStartE2EDuration="6.415833342s" totalImagesPullingTime="5.510029985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:37.412598497 +0000 UTC m=+282.512198643" watchObservedRunningTime="2026-11-03 23:33:37.415833342 +0000 UTC m=+282.515433488"
Nov 03 23:33:38.341180 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:38.341149    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qnvzl"
Nov 03 23:33:40.350987 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:40.350958    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-tbczr"
Nov 03 23:33:48.392563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:48.392530    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 03 23:33:55.036063 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:55.036034    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:33:55.036467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:55.036224    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:33:57.613286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.613251    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"]
Nov 03 23:33:57.618040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.618018    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.627682 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.627656    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8gff7\""
Nov 03 23:33:57.630189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630159    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630202    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630283    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350ff29e-6ece-4954-b340-80ef23b349dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630330    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630355    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350ff29e-6ece-4954-b340-80ef23b349dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630473    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxx9x\" (UniqueName: \"kubernetes.io/projected/350ff29e-6ece-4954-b340-80ef23b349dc-kube-api-access-fxx9x\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630504    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630526    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.630676 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.630566    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350ff29e-6ece-4954-b340-80ef23b349dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.638673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.638647    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"]
Nov 03 23:33:57.731555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731524    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731564    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731589    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350ff29e-6ece-4954-b340-80ef23b349dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731625    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731643    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731658    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350ff29e-6ece-4954-b340-80ef23b349dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731687    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.731777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731713    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350ff29e-6ece-4954-b340-80ef23b349dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.732134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.731826    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxx9x\" (UniqueName: \"kubernetes.io/projected/350ff29e-6ece-4954-b340-80ef23b349dc-kube-api-access-fxx9x\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.732134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.732085    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.732203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.732180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.732291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.732268    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.732355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.732300    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.732516 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.732500    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350ff29e-6ece-4954-b340-80ef23b349dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.734238 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.734215    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350ff29e-6ece-4954-b340-80ef23b349dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.734477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.734459    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350ff29e-6ece-4954-b340-80ef23b349dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.750359 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.750320    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350ff29e-6ece-4954-b340-80ef23b349dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.753218 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.753190    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxx9x\" (UniqueName: \"kubernetes.io/projected/350ff29e-6ece-4954-b340-80ef23b349dc-kube-api-access-fxx9x\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485ggl2s\" (UID: \"350ff29e-6ece-4954-b340-80ef23b349dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:57.931473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:57.931392    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:58.042846 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:33:58.042800    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:33:58.230018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.229991    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"]
Nov 03 23:33:58.231865 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:33:58.231841    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350ff29e_6ece_4954_b340_80ef23b349dc.slice/crio-952a6b467b6c654fb5b616a9bc9543be850940d86b5e4d4897d6d9a004395e4b WatchSource:0}: Error finding container 952a6b467b6c654fb5b616a9bc9543be850940d86b5e4d4897d6d9a004395e4b: Status 404 returned error can't find the container with id 952a6b467b6c654fb5b616a9bc9543be850940d86b5e4d4897d6d9a004395e4b
Nov 03 23:33:58.233968 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.233942    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:33:58.234034 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.234002    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:33:58.234070 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.234032    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:33:58.483991 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.483907    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s" event={"ID":"350ff29e-6ece-4954-b340-80ef23b349dc","Type":"ContainerStarted","Data":"c733b229b0793aec76a8cf99ec4cff4b6be6067b7656de2d900bdbbc358555d5"}
Nov 03 23:33:58.483991 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.483969    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s" event={"ID":"350ff29e-6ece-4954-b340-80ef23b349dc","Type":"ContainerStarted","Data":"952a6b467b6c654fb5b616a9bc9543be850940d86b5e4d4897d6d9a004395e4b"}
Nov 03 23:33:58.523024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.522972    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s" podStartSLOduration=1.52295699 podStartE2EDuration="1.52295699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:58.522151742 +0000 UTC m=+303.621751886" watchObservedRunningTime="2026-11-03 23:33:58.52295699 +0000 UTC m=+303.622557134"
Nov 03 23:33:58.931987 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:58.931911    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:59.531965 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.531935    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wg79x"]
Nov 03 23:33:59.591564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.591524    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wg79x"]
Nov 03 23:33:59.591754 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.591636    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:33:59.601279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.601251    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jjxq7\""
Nov 03 23:33:59.648863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.648830    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glr4x\" (UniqueName: \"kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x\") pod \"authorino-57f7f8659f-wg79x\" (UID: \"d6483d50-5883-4131-abd1-440a20c3351f\") " pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:33:59.750321 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.750286    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glr4x\" (UniqueName: \"kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x\") pod \"authorino-57f7f8659f-wg79x\" (UID: \"d6483d50-5883-4131-abd1-440a20c3351f\") " pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:33:59.763435 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.763402    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glr4x\" (UniqueName: \"kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x\") pod \"authorino-57f7f8659f-wg79x\" (UID: \"d6483d50-5883-4131-abd1-440a20c3351f\") " pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:33:59.937454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.937424    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:33:59.940555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:33:59.940533    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:34:00.235892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:00.235867    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wg79x"]
Nov 03 23:34:00.239111 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:00.239078    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6483d50_5883_4131_abd1_440a20c3351f.slice/crio-6160d26a34d5087b631805cbe359a42490c5520786f60e58b90f6f94d86ca61a WatchSource:0}: Error finding container 6160d26a34d5087b631805cbe359a42490c5520786f60e58b90f6f94d86ca61a: Status 404 returned error can't find the container with id 6160d26a34d5087b631805cbe359a42490c5520786f60e58b90f6f94d86ca61a
Nov 03 23:34:00.240773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:00.240723    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:34:00.496187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:00.496102    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wg79x" event={"ID":"d6483d50-5883-4131-abd1-440a20c3351f","Type":"ContainerStarted","Data":"6160d26a34d5087b631805cbe359a42490c5520786f60e58b90f6f94d86ca61a"}
Nov 03 23:34:00.496333 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:00.496299    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:34:00.497176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:00.497157    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485ggl2s"
Nov 03 23:34:01.393826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.393795    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"]
Nov 03 23:34:01.396592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.396567    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.401485 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.401456    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-sq6qc\""
Nov 03 23:34:01.401649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.401634    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 23:34:01.407142 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.407114    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 23:34:01.407294 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.407151    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 23:34:01.410480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.410459    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"]
Nov 03 23:34:01.467517 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.467483    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/769a6ec6-ceed-444c-aba3-739a7430928d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.467517 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.467518    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/769a6ec6-ceed-444c-aba3-739a7430928d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.467797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.467552    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/769a6ec6-ceed-444c-aba3-739a7430928d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.467797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.467695    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjmh\" (UniqueName: \"kubernetes.io/projected/769a6ec6-ceed-444c-aba3-739a7430928d-kube-api-access-ggjmh\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.569070 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.569032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/769a6ec6-ceed-444c-aba3-739a7430928d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.569070 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.569079    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/769a6ec6-ceed-444c-aba3-739a7430928d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.569338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.569122    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/769a6ec6-ceed-444c-aba3-739a7430928d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.569338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.569205    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjmh\" (UniqueName: \"kubernetes.io/projected/769a6ec6-ceed-444c-aba3-739a7430928d-kube-api-access-ggjmh\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.569591 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.569563    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/769a6ec6-ceed-444c-aba3-739a7430928d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.569832 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.569811    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/769a6ec6-ceed-444c-aba3-739a7430928d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.570506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.570478    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/769a6ec6-ceed-444c-aba3-739a7430928d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.583177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.583155    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjmh\" (UniqueName: \"kubernetes.io/projected/769a6ec6-ceed-444c-aba3-739a7430928d-kube-api-access-ggjmh\") pod \"kserve-module-controller-manager-7566cf5fc8-5hbhg\" (UID: \"769a6ec6-ceed-444c-aba3-739a7430928d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.709594 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.709556    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:01.991484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:01.991406    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"]
Nov 03 23:34:01.994832 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:01.994798    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769a6ec6_ceed_444c_aba3_739a7430928d.slice/crio-1b338005cc311fede1fa112fad3b491b68a701d988dd384d64b13be0d641b823 WatchSource:0}: Error finding container 1b338005cc311fede1fa112fad3b491b68a701d988dd384d64b13be0d641b823: Status 404 returned error can't find the container with id 1b338005cc311fede1fa112fad3b491b68a701d988dd384d64b13be0d641b823
Nov 03 23:34:02.518951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:02.518912    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg" event={"ID":"769a6ec6-ceed-444c-aba3-739a7430928d","Type":"ContainerStarted","Data":"1b338005cc311fede1fa112fad3b491b68a701d988dd384d64b13be0d641b823"}
Nov 03 23:34:03.526506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:03.526459    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wg79x" event={"ID":"d6483d50-5883-4131-abd1-440a20c3351f","Type":"ContainerStarted","Data":"0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592"}
Nov 03 23:34:03.544847 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:03.544787    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wg79x" podStartSLOduration=2.190142029 podStartE2EDuration="4.544767161s" totalImagesPullingTime="2.354625132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:03.544197762 +0000 UTC m=+308.643797906" watchObservedRunningTime="2026-11-03 23:34:03.544767161 +0000 UTC m=+308.644367298"
Nov 03 23:34:04.532354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:04.532319    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg" event={"ID":"769a6ec6-ceed-444c-aba3-739a7430928d","Type":"ContainerStarted","Data":"839d28963efc9f1e5872007a52aeac36fe86f3440124688ce1505dc978e8f783"}
Nov 03 23:34:04.532766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:04.532520    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:04.558009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:04.557956    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg" podStartSLOduration=1.193680641 podStartE2EDuration="3.557942633s" totalImagesPullingTime="2.364261992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:04.556289437 +0000 UTC m=+309.655889582" watchObservedRunningTime="2026-11-03 23:34:04.557942633 +0000 UTC m=+309.657542777"
Nov 03 23:34:06.677399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:06.677325    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wg79x"]
Nov 03 23:34:06.677844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:06.677480    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wg79x" podUID="d6483d50-5883-4131-abd1-440a20c3351f" containerName="authorino" containerID="cri-o://0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592" gracePeriod=30
Nov 03 23:34:06.924129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:06.924105    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:34:07.014883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.014851    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x\" (UniqueName: \"kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x\") pod \"d6483d50-5883-4131-abd1-440a20c3351f\" (UID: \"d6483d50-5883-4131-abd1-440a20c3351f\") "
Nov 03 23:34:07.017007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.016980    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x" pod "d6483d50-5883-4131-abd1-440a20c3351f" (UID: "d6483d50-5883-4131-abd1-440a20c3351f"). InnerVolumeSpecName "kube-api-access-glr4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:34:07.115690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.115655    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glr4x\" (UniqueName: \"kubernetes.io/projected/d6483d50-5883-4131-abd1-440a20c3351f-kube-api-access-glr4x\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:07.543977 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.543941    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:07.544384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.544365    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d6483d50-5883-4131-abd1-440a20c3351f" containerName="authorino"
Nov 03 23:34:07.544473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.544386    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6483d50-5883-4131-abd1-440a20c3351f" containerName="authorino"
Nov 03 23:34:07.544531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.544521    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6483d50-5883-4131-abd1-440a20c3351f" containerName="authorino"
Nov 03 23:34:07.547587 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.547555    5074 generic.go:356] "Generic (PLEG): container finished" podID="d6483d50-5883-4131-abd1-440a20c3351f" containerID="0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592" exitCode=0
Nov 03 23:34:07.547697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.547665    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wg79x"
Nov 03 23:34:07.569621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.569595    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wg79x" event={"ID":"d6483d50-5883-4131-abd1-440a20c3351f","Type":"ContainerDied","Data":"0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592"}
Nov 03 23:34:07.569777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.569633    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:07.569777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.569665    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wg79x" event={"ID":"d6483d50-5883-4131-abd1-440a20c3351f","Type":"ContainerDied","Data":"6160d26a34d5087b631805cbe359a42490c5520786f60e58b90f6f94d86ca61a"}
Nov 03 23:34:07.569777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.569723    5074 scope.go:122] "RemoveContainer" containerID="0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592"
Nov 03 23:34:07.569777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.569759    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.575348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.575324    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wg79x"]
Nov 03 23:34:07.580878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.580859    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pkmb4\""
Nov 03 23:34:07.581209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.581189    5074 scope.go:122] "RemoveContainer" containerID="0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592"
Nov 03 23:34:07.581546 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:34:07.581518    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592\": container with ID starting with 0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592 not found: ID does not exist" containerID="0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592"
Nov 03 23:34:07.581652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.581558    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592"} err="failed to get container status \"0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592\": rpc error: code = NotFound desc = could not find container \"0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592\": container with ID starting with 0348421e1abf95bccc36ef5976857849b5dc4d26b0faec6ce9d154a328caa592 not found: ID does not exist"
Nov 03 23:34:07.584858 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.584836    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wg79x"]
Nov 03 23:34:07.618436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.618405    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file\") pod \"limitador-limitador-74dfcfb894-s8ffl\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.618436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.618439    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzp5l\" (UniqueName: \"kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l\") pod \"limitador-limitador-74dfcfb894-s8ffl\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.645978 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.645944    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:07.662055 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.662027    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 23:34:07.719286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.719255    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file\") pod \"limitador-limitador-74dfcfb894-s8ffl\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.719697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.719293    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzp5l\" (UniqueName: \"kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l\") pod \"limitador-limitador-74dfcfb894-s8ffl\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.719986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.719968    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file\") pod \"limitador-limitador-74dfcfb894-s8ffl\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.732706 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.732676    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzp5l\" (UniqueName: \"kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l\") pod \"limitador-limitador-74dfcfb894-s8ffl\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:07.914714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:07.914638    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:08.221806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:08.221775    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:08.223440 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:08.223409    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f81ba9b_472b_4e8a_b859_eedf5b638ab1.slice/crio-7a9dfe7ad1c4c154c12905f11684ad14c44739916b06cc9fdf3cf727ca4b4dae WatchSource:0}: Error finding container 7a9dfe7ad1c4c154c12905f11684ad14c44739916b06cc9fdf3cf727ca4b4dae: Status 404 returned error can't find the container with id 7a9dfe7ad1c4c154c12905f11684ad14c44739916b06cc9fdf3cf727ca4b4dae
Nov 03 23:34:08.554910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:08.554825    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl" event={"ID":"2f81ba9b-472b-4e8a-b859-eedf5b638ab1","Type":"ContainerStarted","Data":"7a9dfe7ad1c4c154c12905f11684ad14c44739916b06cc9fdf3cf727ca4b4dae"}
Nov 03 23:34:09.015863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:09.015831    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6483d50-5883-4131-abd1-440a20c3351f" path="/var/lib/kubelet/pods/d6483d50-5883-4131-abd1-440a20c3351f/volumes"
Nov 03 23:34:19.607314 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:19.607225    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl" event={"ID":"2f81ba9b-472b-4e8a-b859-eedf5b638ab1","Type":"ContainerStarted","Data":"82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3"}
Nov 03 23:34:19.607697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:19.607340    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:19.627915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:19.627871    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl" podStartSLOduration=1.602986845 podStartE2EDuration="12.627861996s" totalImagesPullingTime="11.024875151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:19.627394662 +0000 UTC m=+324.726994805" watchObservedRunningTime="2026-11-03 23:34:19.627861996 +0000 UTC m=+324.727462139"
Nov 03 23:34:21.318646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:21.318615    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:21.617101 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:21.617019    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl" podUID="2f81ba9b-472b-4e8a-b859-eedf5b638ab1" containerName="limitador" containerID="cri-o://82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3" gracePeriod=30
Nov 03 23:34:22.558795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.558772    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:22.623677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.623641    5074 generic.go:356] "Generic (PLEG): container finished" podID="2f81ba9b-472b-4e8a-b859-eedf5b638ab1" containerID="82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3" exitCode=0
Nov 03 23:34:22.623852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.623687    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"
Nov 03 23:34:22.623852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.623726    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl" event={"ID":"2f81ba9b-472b-4e8a-b859-eedf5b638ab1","Type":"ContainerDied","Data":"82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3"}
Nov 03 23:34:22.623852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.623802    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-s8ffl" event={"ID":"2f81ba9b-472b-4e8a-b859-eedf5b638ab1","Type":"ContainerDied","Data":"7a9dfe7ad1c4c154c12905f11684ad14c44739916b06cc9fdf3cf727ca4b4dae"}
Nov 03 23:34:22.623852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.623825    5074 scope.go:122] "RemoveContainer" containerID="82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3"
Nov 03 23:34:22.632409 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.632391    5074 scope.go:122] "RemoveContainer" containerID="82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3"
Nov 03 23:34:22.632665 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:34:22.632645    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3\": container with ID starting with 82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3 not found: ID does not exist" containerID="82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3"
Nov 03 23:34:22.632728 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.632679    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3"} err="failed to get container status \"82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3\": rpc error: code = NotFound desc = could not find container \"82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3\": container with ID starting with 82c878ef847c84dc1f39794b5274176156516e3661fce3e25f165869ef3698e3 not found: ID does not exist"
Nov 03 23:34:22.672280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.672254    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l\" (UniqueName: \"kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l\") pod \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") "
Nov 03 23:34:22.672381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.672315    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file\" (UniqueName: \"kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file\") pod \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\" (UID: \"2f81ba9b-472b-4e8a-b859-eedf5b638ab1\") "
Nov 03 23:34:22.672684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.672664    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file" pod "2f81ba9b-472b-4e8a-b859-eedf5b638ab1" (UID: "2f81ba9b-472b-4e8a-b859-eedf5b638ab1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:34:22.674300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.674276    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l" pod "2f81ba9b-472b-4e8a-b859-eedf5b638ab1" (UID: "2f81ba9b-472b-4e8a-b859-eedf5b638ab1"). InnerVolumeSpecName "kube-api-access-hzp5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:34:22.773686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.773655    5074 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-config-file\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:22.773686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.773684    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzp5l\" (UniqueName: \"kubernetes.io/projected/2f81ba9b-472b-4e8a-b859-eedf5b638ab1-kube-api-access-hzp5l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:22.950160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.950131    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:22.954494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:22.954467    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-s8ffl"]
Nov 03 23:34:23.015253 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:23.015224    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f81ba9b-472b-4e8a-b859-eedf5b638ab1" path="/var/lib/kubelet/pods/2f81ba9b-472b-4e8a-b859-eedf5b638ab1/volumes"
Nov 03 23:34:25.538461 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.538428    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5hbhg"
Nov 03 23:34:25.588306 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.588275    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"]
Nov 03 23:34:25.588635 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.588623    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2f81ba9b-472b-4e8a-b859-eedf5b638ab1" containerName="limitador"
Nov 03 23:34:25.588685 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.588637    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f81ba9b-472b-4e8a-b859-eedf5b638ab1" containerName="limitador"
Nov 03 23:34:25.588728 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.588719    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f81ba9b-472b-4e8a-b859-eedf5b638ab1" containerName="limitador"
Nov 03 23:34:25.593113 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.593092    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.598244 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.598223    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 23:34:25.598417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.598241    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 23:34:25.598417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.598223    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 23:34:25.598590 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.598497    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dnxnn\""
Nov 03 23:34:25.600849 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.600828    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"]
Nov 03 23:34:25.700633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.700603    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v99r\" (UniqueName: \"kubernetes.io/projected/a1194887-d1c7-4fd0-8d54-cb43a3f60790-kube-api-access-5v99r\") pod \"postgres-7cdbf6dc44-c2d85\" (UID: \"a1194887-d1c7-4fd0-8d54-cb43a3f60790\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.700814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.700661    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1194887-d1c7-4fd0-8d54-cb43a3f60790-data\") pod \"postgres-7cdbf6dc44-c2d85\" (UID: \"a1194887-d1c7-4fd0-8d54-cb43a3f60790\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.801639 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.801553    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v99r\" (UniqueName: \"kubernetes.io/projected/a1194887-d1c7-4fd0-8d54-cb43a3f60790-kube-api-access-5v99r\") pod \"postgres-7cdbf6dc44-c2d85\" (UID: \"a1194887-d1c7-4fd0-8d54-cb43a3f60790\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.801639 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.801594    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1194887-d1c7-4fd0-8d54-cb43a3f60790-data\") pod \"postgres-7cdbf6dc44-c2d85\" (UID: \"a1194887-d1c7-4fd0-8d54-cb43a3f60790\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.802056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.802036    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1194887-d1c7-4fd0-8d54-cb43a3f60790-data\") pod \"postgres-7cdbf6dc44-c2d85\" (UID: \"a1194887-d1c7-4fd0-8d54-cb43a3f60790\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.815771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.815727    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v99r\" (UniqueName: \"kubernetes.io/projected/a1194887-d1c7-4fd0-8d54-cb43a3f60790-kube-api-access-5v99r\") pod \"postgres-7cdbf6dc44-c2d85\" (UID: \"a1194887-d1c7-4fd0-8d54-cb43a3f60790\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:25.907593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:25.907560    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:26.185688 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:26.185654    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"]
Nov 03 23:34:26.187889 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:26.187860    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1194887_d1c7_4fd0_8d54_cb43a3f60790.slice/crio-1d7d42bf489b5e9b1175649d44d8d063149c7be3fe412bdb07d84a766450a042 WatchSource:0}: Error finding container 1d7d42bf489b5e9b1175649d44d8d063149c7be3fe412bdb07d84a766450a042: Status 404 returned error can't find the container with id 1d7d42bf489b5e9b1175649d44d8d063149c7be3fe412bdb07d84a766450a042
Nov 03 23:34:26.644151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:26.644067    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85" event={"ID":"a1194887-d1c7-4fd0-8d54-cb43a3f60790","Type":"ContainerStarted","Data":"1d7d42bf489b5e9b1175649d44d8d063149c7be3fe412bdb07d84a766450a042"}
Nov 03 23:34:28.035819 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:34:28.035760    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:34:32.676294 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:32.676255    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85" event={"ID":"a1194887-d1c7-4fd0-8d54-cb43a3f60790","Type":"ContainerStarted","Data":"cac277d72e9468da0f9cec7c4769eab5b20cb4258e9f6f2c1346b608ed5d0566"}
Nov 03 23:34:32.676713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:32.676321    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:32.697934 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:32.697869    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85" podStartSLOduration=2.104592636 podStartE2EDuration="7.697852403s" totalImagesPullingTime="5.593259767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:32.696119964 +0000 UTC m=+337.795720110" watchObservedRunningTime="2026-11-03 23:34:32.697852403 +0000 UTC m=+337.797452548"
Nov 03 23:34:38.710467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:38.710438    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2d85"
Nov 03 23:34:46.118591 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.118556    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c47496449-xfsb7"]
Nov 03 23:34:46.122218 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.122198    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.147791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.147772    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 23:34:46.147893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.147817    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 23:34:46.149218 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.149200    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 23:34:46.149330 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.149218    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ww46r\""
Nov 03 23:34:46.161830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.161807    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c47496449-xfsb7"]
Nov 03 23:34:46.190625 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.190599    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.190762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.190630    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.190806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.190755    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntv5\" (UniqueName: \"kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.292213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.292184    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntv5\" (UniqueName: \"kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.292402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.292258    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.292402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.292285    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.294868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.294844    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.294967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.294870    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.319221 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.319180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntv5\" (UniqueName: \"kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5\") pod \"maas-controller-c47496449-xfsb7\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") " pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.435507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.435476    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:46.719190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.719163    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c47496449-xfsb7"]
Nov 03 23:34:46.721658 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:46.721625    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4aeaf1_5f96_4598_94e1_707332255551.slice/crio-30fc25e444ecfe271d6ace355a67abde08566171c03800350464085e6ab29a0c WatchSource:0}: Error finding container 30fc25e444ecfe271d6ace355a67abde08566171c03800350464085e6ab29a0c: Status 404 returned error can't find the container with id 30fc25e444ecfe271d6ace355a67abde08566171c03800350464085e6ab29a0c
Nov 03 23:34:46.748211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:46.748179    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c47496449-xfsb7" event={"ID":"8c4aeaf1-5f96-4598-94e1-707332255551","Type":"ContainerStarted","Data":"30fc25e444ecfe271d6ace355a67abde08566171c03800350464085e6ab29a0c"}
Nov 03 23:34:49.776183 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:49.776084    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c47496449-xfsb7" event={"ID":"8c4aeaf1-5f96-4598-94e1-707332255551","Type":"ContainerStarted","Data":"a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb"}
Nov 03 23:34:49.776183 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:49.776180    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:34:49.821281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:49.821222    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-c47496449-xfsb7" podStartSLOduration=1.06783777 podStartE2EDuration="3.821204622s" totalImagesPullingTime="2.753366852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:49.817368012 +0000 UTC m=+354.916968158" watchObservedRunningTime="2026-11-03 23:34:49.821204622 +0000 UTC m=+354.920804769"
Nov 03 23:34:51.079428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.079391    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"]
Nov 03 23:34:51.084594 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.084568    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.095127 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.095103    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 23:34:51.095417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.095375    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 23:34:51.097921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.097694    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"]
Nov 03 23:34:51.140669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.140633    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/bd35424f-d798-46c3-9897-3aa4059bc928-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.140879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.140685    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd35424f-d798-46c3-9897-3aa4059bc928-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.140879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.140810    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqdl\" (UniqueName: \"kubernetes.io/projected/bd35424f-d798-46c3-9897-3aa4059bc928-kube-api-access-pbqdl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.140879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.140858    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd35424f-d798-46c3-9897-3aa4059bc928-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.156806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.156777    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 23:34:51.156806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.156789    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-kj2k8\""
Nov 03 23:34:51.242295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.242258    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd35424f-d798-46c3-9897-3aa4059bc928-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.242502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.242418    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqdl\" (UniqueName: \"kubernetes.io/projected/bd35424f-d798-46c3-9897-3aa4059bc928-kube-api-access-pbqdl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.242502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.242475    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd35424f-d798-46c3-9897-3aa4059bc928-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.242632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.242579    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/bd35424f-d798-46c3-9897-3aa4059bc928-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.242695 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:34:51.242634    5074 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 23:34:51.242769 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:34:51.242697    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd35424f-d798-46c3-9897-3aa4059bc928-tls-cert podName:bd35424f-d798-46c3-9897-3aa4059bc928 nodeName:}" failed. No retries permitted until 2026-11-03 23:34:51.742676317 +0000 UTC m=+356.842276461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/bd35424f-d798-46c3-9897-3aa4059bc928-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-b7bxn" (UID: "bd35424f-d798-46c3-9897-3aa4059bc928") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 23:34:51.243084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.243060    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd35424f-d798-46c3-9897-3aa4059bc928-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.243647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.243624    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/bd35424f-d798-46c3-9897-3aa4059bc928-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.256318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.256288    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqdl\" (UniqueName: \"kubernetes.io/projected/bd35424f-d798-46c3-9897-3aa4059bc928-kube-api-access-pbqdl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.748079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.748043    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd35424f-d798-46c3-9897-3aa4059bc928-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:51.750470 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:51.750449    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd35424f-d798-46c3-9897-3aa4059bc928-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b7bxn\" (UID: \"bd35424f-d798-46c3-9897-3aa4059bc928\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:52.004922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:52.004843    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:34:52.301572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:52.301543    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"]
Nov 03 23:34:52.304027 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:52.304002    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd35424f_d798_46c3_9897_3aa4059bc928.slice/crio-e439c9a3c1b8993496fb4efddc2a74bccc3f0a9760269e3a27eb893b374f3936 WatchSource:0}: Error finding container e439c9a3c1b8993496fb4efddc2a74bccc3f0a9760269e3a27eb893b374f3936: Status 404 returned error can't find the container with id e439c9a3c1b8993496fb4efddc2a74bccc3f0a9760269e3a27eb893b374f3936
Nov 03 23:34:52.790549 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:52.790513    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn" event={"ID":"bd35424f-d798-46c3-9897-3aa4059bc928","Type":"ContainerStarted","Data":"e439c9a3c1b8993496fb4efddc2a74bccc3f0a9760269e3a27eb893b374f3936"}
Nov 03 23:34:55.539607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.538945    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-5s44m"]
Nov 03 23:34:55.545723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.545700    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.550864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.550839    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-5s44m"]
Nov 03 23:34:55.558710 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.558676    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 23:34:55.558848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.558726    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cwvl7\""
Nov 03 23:34:55.578519 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.578495    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"]
Nov 03 23:34:55.589084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.589053    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"]
Nov 03 23:34:55.589292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.589063    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.610784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.609802    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fvh4c"]
Nov 03 23:34:55.613265 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.613248    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.628751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.628376    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 23:34:55.628751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.628503    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-lnwj4\""
Nov 03 23:34:55.628751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.628633    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4mrtt\""
Nov 03 23:34:55.628751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.628701    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 23:34:55.629649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.629631    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fvh4c"]
Nov 03 23:34:55.690671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.690623    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvrt\" (UniqueName: \"kubernetes.io/projected/5bc3460b-b5fc-4efd-816f-fcd07a49b000-kube-api-access-9rvrt\") pod \"kserve-controller-manager-776cc66d47-5s44m\" (UID: \"5bc3460b-b5fc-4efd-816f-fcd07a49b000\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.690969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.690691    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84jx\" (UniqueName: \"kubernetes.io/projected/4a0f933a-4da9-40ef-b486-40abeda40321-kube-api-access-w84jx\") pod \"model-serving-api-75bd89fdfc-fvh4c\" (UID: \"4a0f933a-4da9-40ef-b486-40abeda40321\") " pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.690969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.690896    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f933a-4da9-40ef-b486-40abeda40321-tls-certs\") pod \"model-serving-api-75bd89fdfc-fvh4c\" (UID: \"4a0f933a-4da9-40ef-b486-40abeda40321\") " pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.691085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.690980    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bc3460b-b5fc-4efd-816f-fcd07a49b000-cert\") pod \"kserve-controller-manager-776cc66d47-5s44m\" (UID: \"5bc3460b-b5fc-4efd-816f-fcd07a49b000\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.691085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.691008    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l2xfv\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.691085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.691069    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9kz\" (UniqueName: \"kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz\") pod \"llmisvc-controller-manager-869d5dcc5d-l2xfv\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.792573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.792362    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f933a-4da9-40ef-b486-40abeda40321-tls-certs\") pod \"model-serving-api-75bd89fdfc-fvh4c\" (UID: \"4a0f933a-4da9-40ef-b486-40abeda40321\") " pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.792573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.792415    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bc3460b-b5fc-4efd-816f-fcd07a49b000-cert\") pod \"kserve-controller-manager-776cc66d47-5s44m\" (UID: \"5bc3460b-b5fc-4efd-816f-fcd07a49b000\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.792573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.792445    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l2xfv\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.792573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.792503    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9kz\" (UniqueName: \"kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz\") pod \"llmisvc-controller-manager-869d5dcc5d-l2xfv\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.792573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.792562    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvrt\" (UniqueName: \"kubernetes.io/projected/5bc3460b-b5fc-4efd-816f-fcd07a49b000-kube-api-access-9rvrt\") pod \"kserve-controller-manager-776cc66d47-5s44m\" (UID: \"5bc3460b-b5fc-4efd-816f-fcd07a49b000\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.793058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.792613    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84jx\" (UniqueName: \"kubernetes.io/projected/4a0f933a-4da9-40ef-b486-40abeda40321-kube-api-access-w84jx\") pod \"model-serving-api-75bd89fdfc-fvh4c\" (UID: \"4a0f933a-4da9-40ef-b486-40abeda40321\") " pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.795655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.795625    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bc3460b-b5fc-4efd-816f-fcd07a49b000-cert\") pod \"kserve-controller-manager-776cc66d47-5s44m\" (UID: \"5bc3460b-b5fc-4efd-816f-fcd07a49b000\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.797437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.797398    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f933a-4da9-40ef-b486-40abeda40321-tls-certs\") pod \"model-serving-api-75bd89fdfc-fvh4c\" (UID: \"4a0f933a-4da9-40ef-b486-40abeda40321\") " pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.797437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.797385    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l2xfv\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.812952 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.812892    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvrt\" (UniqueName: \"kubernetes.io/projected/5bc3460b-b5fc-4efd-816f-fcd07a49b000-kube-api-access-9rvrt\") pod \"kserve-controller-manager-776cc66d47-5s44m\" (UID: \"5bc3460b-b5fc-4efd-816f-fcd07a49b000\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.813683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.813633    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84jx\" (UniqueName: \"kubernetes.io/projected/4a0f933a-4da9-40ef-b486-40abeda40321-kube-api-access-w84jx\") pod \"model-serving-api-75bd89fdfc-fvh4c\" (UID: \"4a0f933a-4da9-40ef-b486-40abeda40321\") " pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:55.815255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.815227    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9kz\" (UniqueName: \"kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz\") pod \"llmisvc-controller-manager-869d5dcc5d-l2xfv\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.883054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.882983    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:55.912188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.912160    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:55.933185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:55.933138    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:56.193505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.193470    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-5s44m"]
Nov 03 23:34:56.195795 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:56.195765    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc3460b_b5fc_4efd_816f_fcd07a49b000.slice/crio-715318c01fd4ac4d8cf9286566fc2b14a5ac499ecf960ae453b8dc2ac011e0c2 WatchSource:0}: Error finding container 715318c01fd4ac4d8cf9286566fc2b14a5ac499ecf960ae453b8dc2ac011e0c2: Status 404 returned error can't find the container with id 715318c01fd4ac4d8cf9286566fc2b14a5ac499ecf960ae453b8dc2ac011e0c2
Nov 03 23:34:56.250867 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:56.250833    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod362729fe_07d8_4876_9976_14de8f44a500.slice/crio-3b674aeee9383c7356d154c56895858faf457e4f7f17c87054e104f705aa2d6b WatchSource:0}: Error finding container 3b674aeee9383c7356d154c56895858faf457e4f7f17c87054e104f705aa2d6b: Status 404 returned error can't find the container with id 3b674aeee9383c7356d154c56895858faf457e4f7f17c87054e104f705aa2d6b
Nov 03 23:34:56.251066 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.251044    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"]
Nov 03 23:34:56.291874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.291843    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fvh4c"]
Nov 03 23:34:56.295470 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:56.295439    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0f933a_4da9_40ef_b486_40abeda40321.slice/crio-10e9f20f62ea5f980685361bd7d57718e77d7f07ae08e63a22c0d8f2d7223830 WatchSource:0}: Error finding container 10e9f20f62ea5f980685361bd7d57718e77d7f07ae08e63a22c0d8f2d7223830: Status 404 returned error can't find the container with id 10e9f20f62ea5f980685361bd7d57718e77d7f07ae08e63a22c0d8f2d7223830
Nov 03 23:34:56.671025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.670944    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-4cp2t"]
Nov 03 23:34:56.687877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.687843    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.695044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.689354    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-4cp2t"]
Nov 03 23:34:56.804430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.804156    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.804430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.804202    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.804430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.804302    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wpn\" (UniqueName: \"kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.831787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.820783    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c" event={"ID":"4a0f933a-4da9-40ef-b486-40abeda40321","Type":"ContainerStarted","Data":"10e9f20f62ea5f980685361bd7d57718e77d7f07ae08e63a22c0d8f2d7223830"}
Nov 03 23:34:56.831787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.823324    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m" event={"ID":"5bc3460b-b5fc-4efd-816f-fcd07a49b000","Type":"ContainerStarted","Data":"715318c01fd4ac4d8cf9286566fc2b14a5ac499ecf960ae453b8dc2ac011e0c2"}
Nov 03 23:34:56.831787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.825375    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv" event={"ID":"362729fe-07d8-4876-9976-14de8f44a500","Type":"ContainerStarted","Data":"3b674aeee9383c7356d154c56895858faf457e4f7f17c87054e104f705aa2d6b"}
Nov 03 23:34:56.905429 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.905392    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wpn\" (UniqueName: \"kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.905639 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.905495    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.905639 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.905522    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.908870 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.908836    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.910395 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.910350    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:56.930209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:56.930142    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wpn\" (UniqueName: \"kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn\") pod \"maas-controller-9cc8d5ff6-4cp2t\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") " pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:57.013499 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:57.013272    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:57.347858 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:57.347542    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-4cp2t"]
Nov 03 23:34:57.350667 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:34:57.350597    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9e9ff_13f0_4d3c_ba08_a308ce67d227.slice/crio-c762c085cf446939efe1f0928742c50e904930292a6f9e8ec1660d75b742b461 WatchSource:0}: Error finding container c762c085cf446939efe1f0928742c50e904930292a6f9e8ec1660d75b742b461: Status 404 returned error can't find the container with id c762c085cf446939efe1f0928742c50e904930292a6f9e8ec1660d75b742b461
Nov 03 23:34:57.845981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:57.845936    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" event={"ID":"07f9e9ff-13f0-4d3c-ba08-a308ce67d227","Type":"ContainerStarted","Data":"c762c085cf446939efe1f0928742c50e904930292a6f9e8ec1660d75b742b461"}
Nov 03 23:34:58.051400 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:34:58.051334    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:34:58.856267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:58.856227    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" event={"ID":"07f9e9ff-13f0-4d3c-ba08-a308ce67d227","Type":"ContainerStarted","Data":"27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"}
Nov 03 23:34:58.857136 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:58.857112    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:34:58.880165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:58.880111    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" podStartSLOduration=2.281963263 podStartE2EDuration="2.880094851s" totalImagesPullingTime="598.131588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:58.878983999 +0000 UTC m=+363.978584143" watchObservedRunningTime="2026-11-03 23:34:58.880094851 +0000 UTC m=+363.979694995"
Nov 03 23:34:59.866726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.866288    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv" event={"ID":"362729fe-07d8-4876-9976-14de8f44a500","Type":"ContainerStarted","Data":"9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1"}
Nov 03 23:34:59.866726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.866352    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:34:59.868169 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.868139    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c" event={"ID":"4a0f933a-4da9-40ef-b486-40abeda40321","Type":"ContainerStarted","Data":"d3900ae18d9fd15c40922905b01251e26138d2a2f2d52c32f9e89d116a0d9fe6"}
Nov 03 23:34:59.868305 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.868248    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:34:59.870644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.870606    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m" event={"ID":"5bc3460b-b5fc-4efd-816f-fcd07a49b000","Type":"ContainerStarted","Data":"54c19e97f23ee34bdb222afccb2396512a6a3d08972429fb0cf1c2aa70709925"}
Nov 03 23:34:59.870908 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.870861    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:34:59.905518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.904835    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv" podStartSLOduration=1.457887408 podStartE2EDuration="4.9048162s" totalImagesPullingTime="3.446928792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:59.900949332 +0000 UTC m=+365.000549681" watchObservedRunningTime="2026-11-03 23:34:59.9048162 +0000 UTC m=+365.004416345"
Nov 03 23:34:59.952668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.952571    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c" podStartSLOduration=1.928031015 podStartE2EDuration="4.952556016s" totalImagesPullingTime="3.024525001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:59.950518326 +0000 UTC m=+365.050118470" watchObservedRunningTime="2026-11-03 23:34:59.952556016 +0000 UTC m=+365.052156160"
Nov 03 23:34:59.977900 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:34:59.977832    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m" podStartSLOduration=1.407167609 podStartE2EDuration="4.977815141s" totalImagesPullingTime="3.570647532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:59.975761768 +0000 UTC m=+365.075361915" watchObservedRunningTime="2026-11-03 23:34:59.977815141 +0000 UTC m=+365.077415285"
Nov 03 23:35:00.783649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:00.783616    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:35:05.914826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:05.914786    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn" event={"ID":"bd35424f-d798-46c3-9897-3aa4059bc928","Type":"ContainerStarted","Data":"36e67a72763b260795432bfba909121b07d172ed32b6298c6220176d6c928a91"}
Nov 03 23:35:05.915236 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:05.914912    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:35:05.937547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:05.937500    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn" podStartSLOduration=1.917496938 podStartE2EDuration="14.937487786s" totalImagesPullingTime="13.019990848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:35:05.936582814 +0000 UTC m=+371.036182958" watchObservedRunningTime="2026-11-03 23:35:05.937487786 +0000 UTC m=+371.037087944"
Nov 03 23:35:10.887474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:10.887442    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 03 23:35:10.890123 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:10.890104    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-fvh4c"
Nov 03 23:35:10.957988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:10.957955    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c47496449-xfsb7"]
Nov 03 23:35:10.958204 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:10.958183    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-c47496449-xfsb7" podUID="8c4aeaf1-5f96-4598-94e1-707332255551" containerName="manager" containerID="cri-o://a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb" gracePeriod=30
Nov 03 23:35:11.202006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.201981    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:35:11.272474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.272445    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert\") pod \"8c4aeaf1-5f96-4598-94e1-707332255551\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") "
Nov 03 23:35:11.272643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.272504    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs\") pod \"8c4aeaf1-5f96-4598-94e1-707332255551\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") "
Nov 03 23:35:11.272643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.272560    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5\" (UniqueName: \"kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5\") pod \"8c4aeaf1-5f96-4598-94e1-707332255551\" (UID: \"8c4aeaf1-5f96-4598-94e1-707332255551\") "
Nov 03 23:35:11.274681 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.274648    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert" pod "8c4aeaf1-5f96-4598-94e1-707332255551" (UID: "8c4aeaf1-5f96-4598-94e1-707332255551"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:35:11.274823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.274693    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5" pod "8c4aeaf1-5f96-4598-94e1-707332255551" (UID: "8c4aeaf1-5f96-4598-94e1-707332255551"). InnerVolumeSpecName "kube-api-access-zntv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:35:11.274823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.274727    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs" pod "8c4aeaf1-5f96-4598-94e1-707332255551" (UID: "8c4aeaf1-5f96-4598-94e1-707332255551"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:35:11.373638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.373600    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntv5\" (UniqueName: \"kubernetes.io/projected/8c4aeaf1-5f96-4598-94e1-707332255551-kube-api-access-zntv5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:35:11.373638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.373640    5074 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-webhook-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:35:11.373849 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.373656    5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4aeaf1-5f96-4598-94e1-707332255551-metrics-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:35:11.948490 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.948454    5074 generic.go:356] "Generic (PLEG): container finished" podID="8c4aeaf1-5f96-4598-94e1-707332255551" containerID="a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb" exitCode=0
Nov 03 23:35:11.948986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.948508    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c47496449-xfsb7"
Nov 03 23:35:11.948986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.948535    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c47496449-xfsb7" event={"ID":"8c4aeaf1-5f96-4598-94e1-707332255551","Type":"ContainerDied","Data":"a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb"}
Nov 03 23:35:11.948986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.948590    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c47496449-xfsb7" event={"ID":"8c4aeaf1-5f96-4598-94e1-707332255551","Type":"ContainerDied","Data":"30fc25e444ecfe271d6ace355a67abde08566171c03800350464085e6ab29a0c"}
Nov 03 23:35:11.948986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.948612    5074 scope.go:122] "RemoveContainer" containerID="a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb"
Nov 03 23:35:11.957836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.957820    5074 scope.go:122] "RemoveContainer" containerID="a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb"
Nov 03 23:35:11.958102 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:35:11.958086    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb\": container with ID starting with a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb not found: ID does not exist" containerID="a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb"
Nov 03 23:35:11.958155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.958112    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb"} err="failed to get container status \"a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb\": rpc error: code = NotFound desc = could not find container \"a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb\": container with ID starting with a8ab1e53f3a6a9a54012be2db328817735dd519a3f40bd9de96a1311494307bb not found: ID does not exist"
Nov 03 23:35:11.973006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.972986    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c47496449-xfsb7"]
Nov 03 23:35:11.978418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:11.978398    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c47496449-xfsb7"]
Nov 03 23:35:13.015938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:13.015908    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4aeaf1-5f96-4598-94e1-707332255551" path="/var/lib/kubelet/pods/8c4aeaf1-5f96-4598-94e1-707332255551/volumes"
Nov 03 23:35:16.926876 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:16.926848    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b7bxn"
Nov 03 23:35:28.019647 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:35:28.019610    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:35:30.888083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:30.888053    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:35:30.890514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:30.890486    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-5s44m"
Nov 03 23:35:39.134944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:39.134904    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:35:39.135466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:39.134989    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:35:41.086135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:41.086044    5074 generic.go:356] "Generic (PLEG): container finished" podID="c332db91-1798-471f-87d8-55217be24115" containerID="6a55d18e08d63d1cf9178955e9b9aca3694ccec6700e6e44cfd0ca08e5c9cb24" exitCode=0
Nov 03 23:35:41.086135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:41.086110    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" event={"ID":"c332db91-1798-471f-87d8-55217be24115","Type":"ContainerDied","Data":"6a55d18e08d63d1cf9178955e9b9aca3694ccec6700e6e44cfd0ca08e5c9cb24"}
Nov 03 23:35:41.086557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:41.086471    5074 scope.go:122] "RemoveContainer" containerID="6a55d18e08d63d1cf9178955e9b9aca3694ccec6700e6e44cfd0ca08e5c9cb24"
Nov 03 23:35:41.087509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:41.087483    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:35:41.087559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:41.087551    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:35:41.087593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:41.087580    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:35:42.092290 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:42.092254    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" event={"ID":"c332db91-1798-471f-87d8-55217be24115","Type":"ContainerStarted","Data":"e09a679e75f6db9114ef81e2a5666408cd689dfda01b20f1dc3086b3daf63665"}
Nov 03 23:35:43.020764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.020240    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 03 23:35:43.020764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.020305    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 03 23:35:43.027882 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.027860    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:35:43.937526 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.937489    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lm9dr"]
Nov 03 23:35:43.937983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.937967    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8c4aeaf1-5f96-4598-94e1-707332255551" containerName="manager"
Nov 03 23:35:43.938029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.937984    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4aeaf1-5f96-4598-94e1-707332255551" containerName="manager"
Nov 03 23:35:43.938118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.938106    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4aeaf1-5f96-4598-94e1-707332255551" containerName="manager"
Nov 03 23:35:43.941438 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.941421    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:43.949729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.949706    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jjxq7\""
Nov 03 23:35:43.949841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.949714    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 23:35:43.951704 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:43.951680    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lm9dr"]
Nov 03 23:35:44.022555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.022518    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 03 23:35:44.022838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.022807    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 03 23:35:44.028392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.028362    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert\") pod \"authorino-6889cbd8c4-lm9dr\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") " pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.028607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.028584    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnwk\" (UniqueName: \"kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk\") pod \"authorino-6889cbd8c4-lm9dr\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") " pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.129390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.129357    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert\") pod \"authorino-6889cbd8c4-lm9dr\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") " pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.129632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.129619    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnwk\" (UniqueName: \"kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk\") pod \"authorino-6889cbd8c4-lm9dr\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") " pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.131915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.131892    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert\") pod \"authorino-6889cbd8c4-lm9dr\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") " pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.143760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.143706    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnwk\" (UniqueName: \"kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk\") pod \"authorino-6889cbd8c4-lm9dr\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") " pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.215481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.214859    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lm9dr"]
Nov 03 23:35:44.215481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.215157    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:44.523828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:44.523698    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lm9dr"]
Nov 03 23:35:44.530473 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:35:44.528973    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd55ddd8_c753_4a14_965c_66074a75ad37.slice/crio-bd636d6a9b91af49261a5dcd0bb89c1ae4ec9f04c67c81140641a8cbba3073de WatchSource:0}: Error finding container bd636d6a9b91af49261a5dcd0bb89c1ae4ec9f04c67c81140641a8cbba3073de: Status 404 returned error can't find the container with id bd636d6a9b91af49261a5dcd0bb89c1ae4ec9f04c67c81140641a8cbba3073de
Nov 03 23:35:45.018763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:45.018607    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 03 23:35:45.018763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:45.018666    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 03 23:35:45.135777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:45.132728    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr" event={"ID":"dd55ddd8-c753-4a14-965c-66074a75ad37","Type":"ContainerStarted","Data":"bd636d6a9b91af49261a5dcd0bb89c1ae4ec9f04c67c81140641a8cbba3073de"}
Nov 03 23:35:46.013572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.013151    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 03 23:35:46.013572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.013213    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 03 23:35:46.141712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.141674    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr" event={"ID":"dd55ddd8-c753-4a14-965c-66074a75ad37","Type":"ContainerStarted","Data":"0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02"}
Nov 03 23:35:46.142104 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.141921    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr" podUID="dd55ddd8-c753-4a14-965c-66074a75ad37" containerName="authorino" containerID="cri-o://0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02" gracePeriod=30
Nov 03 23:35:46.170043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.169988    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr" podStartSLOduration=2.112504935 podStartE2EDuration="3.169974279s" totalImagesPullingTime="1.057469344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:35:46.168990852 +0000 UTC m=+411.268590995" watchObservedRunningTime="2026-11-03 23:35:46.169974279 +0000 UTC m=+411.269574422"
Nov 03 23:35:46.426164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.426127    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:46.452497 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.452471    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert\" (UniqueName: \"kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert\") pod \"dd55ddd8-c753-4a14-965c-66074a75ad37\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") "
Nov 03 23:35:46.453007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.452988    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk\" (UniqueName: \"kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk\") pod \"dd55ddd8-c753-4a14-965c-66074a75ad37\" (UID: \"dd55ddd8-c753-4a14-965c-66074a75ad37\") "
Nov 03 23:35:46.455080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.455038    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk" pod "dd55ddd8-c753-4a14-965c-66074a75ad37" (UID: "dd55ddd8-c753-4a14-965c-66074a75ad37"). InnerVolumeSpecName "kube-api-access-nbnwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:35:46.465235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.465211    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert" pod "dd55ddd8-c753-4a14-965c-66074a75ad37" (UID: "dd55ddd8-c753-4a14-965c-66074a75ad37"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:35:46.554809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.554775    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnwk\" (UniqueName: \"kubernetes.io/projected/dd55ddd8-c753-4a14-965c-66074a75ad37-kube-api-access-nbnwk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:35:46.554809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:46.554806    5074 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd55ddd8-c753-4a14-965c-66074a75ad37-tls-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:35:47.006859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.006817    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 03 23:35:47.007035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.006886    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 03 23:35:47.147668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.147636    5074 generic.go:356] "Generic (PLEG): container finished" podID="dd55ddd8-c753-4a14-965c-66074a75ad37" containerID="0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02" exitCode=0
Nov 03 23:35:47.148195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.147685    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr"
Nov 03 23:35:47.148195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.147725    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr" event={"ID":"dd55ddd8-c753-4a14-965c-66074a75ad37","Type":"ContainerDied","Data":"0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02"}
Nov 03 23:35:47.148195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.147791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lm9dr" event={"ID":"dd55ddd8-c753-4a14-965c-66074a75ad37","Type":"ContainerDied","Data":"bd636d6a9b91af49261a5dcd0bb89c1ae4ec9f04c67c81140641a8cbba3073de"}
Nov 03 23:35:47.148195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.147807    5074 scope.go:122] "RemoveContainer" containerID="0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02"
Nov 03 23:35:47.156483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.156458    5074 scope.go:122] "RemoveContainer" containerID="0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02"
Nov 03 23:35:47.156714 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:35:47.156694    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02\": container with ID starting with 0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02 not found: ID does not exist" containerID="0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02"
Nov 03 23:35:47.156819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.156722    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02"} err="failed to get container status \"0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02\": rpc error: code = NotFound desc = could not find container \"0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02\": container with ID starting with 0df15975f7584dbaa064a0d3963013a714dd359a8452b09702aa66cdee354a02 not found: ID does not exist"
Nov 03 23:35:47.191701 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.191677    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lm9dr"]
Nov 03 23:35:47.218798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:47.218765    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lm9dr"]
Nov 03 23:35:48.010267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:48.010239    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:35:48.010655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:48.010637    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:35:48.011296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:48.011276    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb"
Nov 03 23:35:49.015924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:35:49.015893    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd55ddd8-c753-4a14-965c-66074a75ad37" path="/var/lib/kubelet/pods/dd55ddd8-c753-4a14-965c-66074a75ad37/volumes"
Nov 03 23:35:58.019483 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:35:58.019449    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:36:07.257251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.257220    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"]
Nov 03 23:36:07.257673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.257659    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dd55ddd8-c753-4a14-965c-66074a75ad37" containerName="authorino"
Nov 03 23:36:07.257727 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.257674    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd55ddd8-c753-4a14-965c-66074a75ad37" containerName="authorino"
Nov 03 23:36:07.257786 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.257776    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd55ddd8-c753-4a14-965c-66074a75ad37" containerName="authorino"
Nov 03 23:36:07.260799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.260780    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.269922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.269886    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"]
Nov 03 23:36:07.333518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.333493    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f97340-55be-4356-80cd-123779b1de3c-cert\") pod \"llmisvc-controller-manager-6466c554d8-f5pqr\" (UID: \"74f97340-55be-4356-80cd-123779b1de3c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.333669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.333539    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668p9\" (UniqueName: \"kubernetes.io/projected/74f97340-55be-4356-80cd-123779b1de3c-kube-api-access-668p9\") pod \"llmisvc-controller-manager-6466c554d8-f5pqr\" (UID: \"74f97340-55be-4356-80cd-123779b1de3c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.434697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.434664    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668p9\" (UniqueName: \"kubernetes.io/projected/74f97340-55be-4356-80cd-123779b1de3c-kube-api-access-668p9\") pod \"llmisvc-controller-manager-6466c554d8-f5pqr\" (UID: \"74f97340-55be-4356-80cd-123779b1de3c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.434855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.434774    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f97340-55be-4356-80cd-123779b1de3c-cert\") pod \"llmisvc-controller-manager-6466c554d8-f5pqr\" (UID: \"74f97340-55be-4356-80cd-123779b1de3c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.437258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.437236    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f97340-55be-4356-80cd-123779b1de3c-cert\") pod \"llmisvc-controller-manager-6466c554d8-f5pqr\" (UID: \"74f97340-55be-4356-80cd-123779b1de3c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.448226 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.448199    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668p9\" (UniqueName: \"kubernetes.io/projected/74f97340-55be-4356-80cd-123779b1de3c-kube-api-access-668p9\") pod \"llmisvc-controller-manager-6466c554d8-f5pqr\" (UID: \"74f97340-55be-4356-80cd-123779b1de3c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.574583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.574517    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:07.860623 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:07.860598    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"]
Nov 03 23:36:07.862508 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:36:07.862480    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f97340_55be_4356_80cd_123779b1de3c.slice/crio-3d477f5ae43ed3b51b6ef2c43b5b57a2c39b8e5f7525a3d8b6c51d20bd6d6bc8 WatchSource:0}: Error finding container 3d477f5ae43ed3b51b6ef2c43b5b57a2c39b8e5f7525a3d8b6c51d20bd6d6bc8: Status 404 returned error can't find the container with id 3d477f5ae43ed3b51b6ef2c43b5b57a2c39b8e5f7525a3d8b6c51d20bd6d6bc8
Nov 03 23:36:08.251202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:08.251172    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr" event={"ID":"74f97340-55be-4356-80cd-123779b1de3c","Type":"ContainerStarted","Data":"3d477f5ae43ed3b51b6ef2c43b5b57a2c39b8e5f7525a3d8b6c51d20bd6d6bc8"}
Nov 03 23:36:09.257490 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:09.257455    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr" event={"ID":"74f97340-55be-4356-80cd-123779b1de3c","Type":"ContainerStarted","Data":"8e8f5ed7db4f3d4b3f784726839ed544506eaa2e8a71caf5dd005ae89a659b91"}
Nov 03 23:36:09.258010 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:09.257521    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:09.280133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:09.280085    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr" podStartSLOduration=1.9133507029999999 podStartE2EDuration="2.280072306s" totalImagesPullingTime="366.721603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:36:09.27805426 +0000 UTC m=+434.377654403" watchObservedRunningTime="2026-11-03 23:36:09.280072306 +0000 UTC m=+434.379672446"
Nov 03 23:36:28.020870 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:36:28.020831    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:36:40.264701 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.264671    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f5pqr"
Nov 03 23:36:40.324161 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.324131    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"]
Nov 03 23:36:40.324341 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.324321    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv" podUID="362729fe-07d8-4876-9976-14de8f44a500" containerName="manager" containerID="cri-o://9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1" gracePeriod=30
Nov 03 23:36:40.577589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.577559    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:36:40.603054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.603024    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert\" (UniqueName: \"kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert\") pod \"362729fe-07d8-4876-9976-14de8f44a500\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") "
Nov 03 23:36:40.603224 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.603067    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz\" (UniqueName: \"kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz\") pod \"362729fe-07d8-4876-9976-14de8f44a500\" (UID: \"362729fe-07d8-4876-9976-14de8f44a500\") "
Nov 03 23:36:40.605252 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.605206    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert" pod "362729fe-07d8-4876-9976-14de8f44a500" (UID: "362729fe-07d8-4876-9976-14de8f44a500"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:36:40.605610 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.605431    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz" pod "362729fe-07d8-4876-9976-14de8f44a500" (UID: "362729fe-07d8-4876-9976-14de8f44a500"). InnerVolumeSpecName "kube-api-access-5b9kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:36:40.704065 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.704027    5074 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/362729fe-07d8-4876-9976-14de8f44a500-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:36:40.704065 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:40.704058    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9kz\" (UniqueName: \"kubernetes.io/projected/362729fe-07d8-4876-9976-14de8f44a500-kube-api-access-5b9kz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:36:41.416209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.416129    5074 generic.go:356] "Generic (PLEG): container finished" podID="362729fe-07d8-4876-9976-14de8f44a500" containerID="9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1" exitCode=0
Nov 03 23:36:41.416209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.416178    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"
Nov 03 23:36:41.416652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.416219    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv" event={"ID":"362729fe-07d8-4876-9976-14de8f44a500","Type":"ContainerDied","Data":"9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1"}
Nov 03 23:36:41.416652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.416274    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv" event={"ID":"362729fe-07d8-4876-9976-14de8f44a500","Type":"ContainerDied","Data":"3b674aeee9383c7356d154c56895858faf457e4f7f17c87054e104f705aa2d6b"}
Nov 03 23:36:41.416652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.416303    5074 scope.go:122] "RemoveContainer" containerID="9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1"
Nov 03 23:36:41.425034 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.425014    5074 scope.go:122] "RemoveContainer" containerID="9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1"
Nov 03 23:36:41.425283 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:36:41.425260    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1\": container with ID starting with 9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1 not found: ID does not exist" containerID="9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1"
Nov 03 23:36:41.425344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.425296    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1"} err="failed to get container status \"9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1\": rpc error: code = NotFound desc = could not find container \"9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1\": container with ID starting with 9e4133e53302017155184e9d3debf7e7a17291fd261dcf5388b123a344356dd1 not found: ID does not exist"
Nov 03 23:36:41.436782 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.436760    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"]
Nov 03 23:36:41.441162 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:41.441142    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2xfv"]
Nov 03 23:36:43.015880 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:36:43.015852    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362729fe-07d8-4876-9976-14de8f44a500" path="/var/lib/kubelet/pods/362729fe-07d8-4876-9976-14de8f44a500/volumes"
Nov 03 23:36:58.020508 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:36:58.020469    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:37:28.020243 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:37:28.020163    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:37:58.020449 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:37:58.020404    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:38:02.342953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.342921    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"]
Nov 03 23:38:02.343336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.343322    5074 cpu_manager.go:414] "containerMap: removing container" podUID="362729fe-07d8-4876-9976-14de8f44a500" containerName="manager"
Nov 03 23:38:02.343382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.343336    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362729fe-07d8-4876-9976-14de8f44a500" containerName="manager"
Nov 03 23:38:02.343438 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.343429    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="362729fe-07d8-4876-9976-14de8f44a500" containerName="manager"
Nov 03 23:38:02.346735 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.346711    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.375407 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.375380    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"]
Nov 03 23:38:02.436110 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436074    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436116    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436136    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436205    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436299    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436376    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436430    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h265k\" (UniqueName: \"kubernetes.io/projected/eda3869d-7ca3-4325-b816-dd5e5153ba6c-kube-api-access-h265k\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.436519 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.436496    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537317    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537356    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537478    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h265k\" (UniqueName: \"kubernetes.io/projected/eda3869d-7ca3-4325-b816-dd5e5153ba6c-kube-api-access-h265k\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537517    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537540    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537571    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537600    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537625    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537704    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537709    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537776    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537873    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.537985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.537896    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.538414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.538397    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.539914 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.539888    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.540245 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.540224    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.550450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.550423    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3869d-7ca3-4325-b816-dd5e5153ba6c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.550532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.550482    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h265k\" (UniqueName: \"kubernetes.io/projected/eda3869d-7ca3-4325-b816-dd5e5153ba6c-kube-api-access-h265k\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m6wnb\" (UID: \"eda3869d-7ca3-4325-b816-dd5e5153ba6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.660586 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.660517    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:02.942172 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.942141    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"]
Nov 03 23:38:02.944050 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:38:02.944022    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda3869d_7ca3_4325_b816_dd5e5153ba6c.slice/crio-7668f2c0312029c92078924a03ad94383d226ee4edfd22864ad5e2bba8a9c300 WatchSource:0}: Error finding container 7668f2c0312029c92078924a03ad94383d226ee4edfd22864ad5e2bba8a9c300: Status 404 returned error can't find the container with id 7668f2c0312029c92078924a03ad94383d226ee4edfd22864ad5e2bba8a9c300
Nov 03 23:38:02.946126 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.946097    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:38:02.946192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.946159    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:38:02.946192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:02.946186    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:38:03.848301 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:03.848270    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" event={"ID":"eda3869d-7ca3-4325-b816-dd5e5153ba6c","Type":"ContainerStarted","Data":"19e6b2f0403976aba463a18d718131ea24bc28bf87a37c95d0b7e46a7cc2c1fc"}
Nov 03 23:38:03.848685 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:03.848320    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" event={"ID":"eda3869d-7ca3-4325-b816-dd5e5153ba6c","Type":"ContainerStarted","Data":"7668f2c0312029c92078924a03ad94383d226ee4edfd22864ad5e2bba8a9c300"}
Nov 03 23:38:03.878904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:03.878856    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podStartSLOduration=1.87884267 podStartE2EDuration="1.87884267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:03.876123775 +0000 UTC m=+548.975723919" watchObservedRunningTime="2026-11-03 23:38:03.87884267 +0000 UTC m=+548.978442815"
Nov 03 23:38:04.661336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:04.661305    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:04.663054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:04.663029    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:38:04.663159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:04.663077    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 23:38:05.661463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:05.661426    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:38:05.661895 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:05.661496    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 23:38:06.661712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:06.661674    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:38:06.662138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:06.661769    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 23:38:07.660866 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:07.660832    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:38:07.661039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:07.660889    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 23:38:08.661030 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:08.660996    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:38:08.661417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:08.661049    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 23:38:09.661427 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:09.661383    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:38:09.661864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:09.661456    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 23:38:11.261145 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:11.261107    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:11.262266 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:11.262248    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:11.263045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:11.263015    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb"
Nov 03 23:38:17.368504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.368461    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"]
Nov 03 23:38:17.373173 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.373147    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.388050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.388022    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"]
Nov 03 23:38:17.502108 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502058    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502317 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502135    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502317 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502168    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502325    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9lw\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502367    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502449    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502508    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502532    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.502642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.502601    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.603534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603496    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.603860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603561    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.603860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603588    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.603860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603648    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9lw\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.603860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603678    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603917    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.603937    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604006    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604027    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604050    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604337 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604119    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604337 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604286    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604513 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604491    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.604660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.604637    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.606316 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.606287    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.606550 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.606527    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.617016 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.616991    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.617379 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.617358    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9lw\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-d574r\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.692998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.692973    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:17.976194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.976171    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"]
Nov 03 23:38:17.977567 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:38:17.977539    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88df79fa_44f6_4a0d_8370_adae26d56a88.slice/crio-60f8afe73d6232ae4ccefb6eb8879667123c038f9c47908fcdd973adbde72d3f WatchSource:0}: Error finding container 60f8afe73d6232ae4ccefb6eb8879667123c038f9c47908fcdd973adbde72d3f: Status 404 returned error can't find the container with id 60f8afe73d6232ae4ccefb6eb8879667123c038f9c47908fcdd973adbde72d3f
Nov 03 23:38:17.979583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.979552    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:38:17.979669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.979628    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:38:17.979728 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:17.979671    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:38:18.930252 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:18.930213    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" event={"ID":"88df79fa-44f6-4a0d-8370-adae26d56a88","Type":"ContainerStarted","Data":"d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c"}
Nov 03 23:38:18.930673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:18.930284    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" event={"ID":"88df79fa-44f6-4a0d-8370-adae26d56a88","Type":"ContainerStarted","Data":"60f8afe73d6232ae4ccefb6eb8879667123c038f9c47908fcdd973adbde72d3f"}
Nov 03 23:38:18.959141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:18.959091    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podStartSLOduration=1.9590772269999999 podStartE2EDuration="1.959077227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:18.957527895 +0000 UTC m=+564.057128068" watchObservedRunningTime="2026-11-03 23:38:18.959077227 +0000 UTC m=+564.058677370"
Nov 03 23:38:19.694028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:19.693994    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:19.695769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:19.695724    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:38:19.695989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:19.695783    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 23:38:20.693570 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:20.693533    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:38:20.693979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:20.693585    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 23:38:21.694010 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:21.693968    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:38:21.694422 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:21.694031    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 23:38:22.694139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:22.694102    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:38:22.694608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:22.694172    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 23:38:23.693687 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:23.693651    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:38:23.693867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:23.693724    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 23:38:24.693792 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:24.693755    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:38:24.694320 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:24.693820    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 23:38:26.286535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:26.286501    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:26.287001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:26.286975    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:26.287716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:26.287694    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 03 23:38:28.020219 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:38:28.020185    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:38:32.429851 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:32.429803    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" podUID="14f459d1824c1ea630aa90a99306c647" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:38:55.068765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:55.068721    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:38:55.069920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:38:55.069899    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:38:58.020508 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:38:58.020468    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:39:26.178460 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.178420    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"]
Nov 03 23:39:26.183171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.183148    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.197715 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.197656    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-044f4dcd\""
Nov 03 23:39:26.198286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.198262    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"]
Nov 03 23:39:26.203025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.202999    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"]
Nov 03 23:39:26.203142 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.203101    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.229590 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.229565    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-044f4dcd-dockercfg-pn7pr\""
Nov 03 23:39:26.260811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.260779    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"]
Nov 03 23:39:26.288299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.288258    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.288481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.288324    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4l8\" (UniqueName: \"kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8\") pod \"payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.288481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.288347    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrtm\" (UniqueName: \"kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm\") pod \"payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") " pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.288481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.288377    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") " pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.389135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.389099    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrtm\" (UniqueName: \"kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm\") pod \"payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") " pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.389329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.389157    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") " pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.389329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.389278    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.389454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.389333    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4l8\" (UniqueName: \"kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8\") pod \"payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.390065 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.390037    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") " pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.390277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.390255    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.415018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.414983    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrtm\" (UniqueName: \"kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm\") pod \"payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") " pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.415153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.415101    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4l8\" (UniqueName: \"kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8\") pod \"payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.505512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.505476    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:26.516941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.516911    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:26.791950 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.791927    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"]
Nov 03 23:39:26.793954 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:26.793928    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a451397_4b83_4a5e_ad65_f64f1071b288.slice/crio-1a870093e2ec792f8d6bc2e1735368c5c80ad93dcb44499cfda1a6de770bae25 WatchSource:0}: Error finding container 1a870093e2ec792f8d6bc2e1735368c5c80ad93dcb44499cfda1a6de770bae25: Status 404 returned error can't find the container with id 1a870093e2ec792f8d6bc2e1735368c5c80ad93dcb44499cfda1a6de770bae25
Nov 03 23:39:26.795614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.795598    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:39:26.811100 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:26.811077    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"]
Nov 03 23:39:26.812885 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:26.812860    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a69959b_0fc0_4d56_a300_9453c34644a3.slice/crio-93827488360e9cb54e6425f38df38568e9c897214987e087614516647162146e WatchSource:0}: Error finding container 93827488360e9cb54e6425f38df38568e9c897214987e087614516647162146e: Status 404 returned error can't find the container with id 93827488360e9cb54e6425f38df38568e9c897214987e087614516647162146e
Nov 03 23:39:27.288271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:27.288242    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl" event={"ID":"7a69959b-0fc0-4d56-a300-9453c34644a3","Type":"ContainerStarted","Data":"93827488360e9cb54e6425f38df38568e9c897214987e087614516647162146e"}
Nov 03 23:39:27.289735 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:27.289708    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl" event={"ID":"9a451397-4b83-4a5e-ad65-f64f1071b288","Type":"ContainerStarted","Data":"1a870093e2ec792f8d6bc2e1735368c5c80ad93dcb44499cfda1a6de770bae25"}
Nov 03 23:39:28.035091 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:39:28.035047    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:39:28.595775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.595726    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"]
Nov 03 23:39:28.613806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.613693    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.615127 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.615101    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"]
Nov 03 23:39:28.623550 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.623530    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:39:28.623672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.623604    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:39:28.722707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.722672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.722920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.722776    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmqw\" (UniqueName: \"kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.722920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.722832    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.826201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.824444    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmqw\" (UniqueName: \"kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.826201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.824518    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.826201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.824595    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.826201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.825451    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.841475 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.841443    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.845165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.845147    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmqw\" (UniqueName: \"kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw\") pod \"maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:28.967994 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:28.967957    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:29.310988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:29.310939    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"]
Nov 03 23:39:30.323826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.323775    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl" event={"ID":"7a69959b-0fc0-4d56-a300-9453c34644a3","Type":"ContainerStarted","Data":"2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add"}
Nov 03 23:39:30.324334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.323985    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:30.325342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.325318    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl" event={"ID":"9a451397-4b83-4a5e-ad65-f64f1071b288","Type":"ContainerStarted","Data":"7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed"}
Nov 03 23:39:30.325457 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.325374    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:30.326553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.326525    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8" event={"ID":"d33517df-f607-41a6-98df-ee77c5d4496c","Type":"ContainerStarted","Data":"118cddc4bc92780bf2fbb1f8e4e9940edefb54a2eaa581b57b819ec51cdd5ac8"}
Nov 03 23:39:30.359047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.358985    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl" podStartSLOduration=1.785283861 podStartE2EDuration="4.358967118s" totalImagesPullingTime="2.573683257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:30.355582284 +0000 UTC m=+635.455182428" watchObservedRunningTime="2026-11-03 23:39:30.358967118 +0000 UTC m=+635.458567262"
Nov 03 23:39:30.389415 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:30.389357    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl" podStartSLOduration=1.671378404 podStartE2EDuration="4.389340987s" totalImagesPullingTime="2.717962583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:30.385425739 +0000 UTC m=+635.485025909" watchObservedRunningTime="2026-11-03 23:39:30.389340987 +0000 UTC m=+635.488941131"
Nov 03 23:39:34.275577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.275535    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"]
Nov 03 23:39:34.279176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.279154    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.307188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.307153    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"]
Nov 03 23:39:34.391694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.391659    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.391918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.391716    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxmb\" (UniqueName: \"kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.391918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.391839    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.499990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.493218    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.499990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.493319    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxmb\" (UniqueName: \"kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.499990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.493370    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.499990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.496206    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.499990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.498028    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.567330 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.567256    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxmb\" (UniqueName: \"kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb\") pod \"maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.593094 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.593064    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:34.781751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.781701    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"]
Nov 03 23:39:34.786628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.786608    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:34.831232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.831139    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d3157179-dockercfg-8qn7l\""
Nov 03 23:39:34.832515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.832492    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d3157179\""
Nov 03 23:39:34.845849 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.845806    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"]
Nov 03 23:39:34.897213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.897182    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:34.897372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.897291    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5hm\" (UniqueName: \"kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm\") pod \"payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:34.952474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.952444    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"]
Nov 03 23:39:34.954148 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:34.954123    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d10a2f8_a8d4_4b96_9c22_c80f32d90cd6.slice/crio-1652c12774a5bcdcfce84f32772d94f5731e7ed84a8da84da65ff25a46b4bdd6 WatchSource:0}: Error finding container 1652c12774a5bcdcfce84f32772d94f5731e7ed84a8da84da65ff25a46b4bdd6: Status 404 returned error can't find the container with id 1652c12774a5bcdcfce84f32772d94f5731e7ed84a8da84da65ff25a46b4bdd6
Nov 03 23:39:34.998278 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.998256    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:34.998390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.998318    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5hm\" (UniqueName: \"kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm\") pod \"payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:34.998990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:34.998974    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:35.045103 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:35.045072    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5hm\" (UniqueName: \"kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm\") pod \"payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:35.100058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:35.099938    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:35.354672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:35.354580    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8" event={"ID":"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6","Type":"ContainerStarted","Data":"1652c12774a5bcdcfce84f32772d94f5731e7ed84a8da84da65ff25a46b4bdd6"}
Nov 03 23:39:35.391989 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:35.391943    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf518a4a_b7ad_4683_b1b7_f8f2c25f6bae.slice/crio-79a31cfaa05ad3b80231a6a877e248da49fc65efc7c6d967b5ed22be1634db02 WatchSource:0}: Error finding container 79a31cfaa05ad3b80231a6a877e248da49fc65efc7c6d967b5ed22be1634db02: Status 404 returned error can't find the container with id 79a31cfaa05ad3b80231a6a877e248da49fc65efc7c6d967b5ed22be1634db02
Nov 03 23:39:35.393264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:35.393237    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"]
Nov 03 23:39:36.364688 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:36.364646    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5" event={"ID":"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae","Type":"ContainerStarted","Data":"03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603"}
Nov 03 23:39:36.365134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:36.364716    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5" event={"ID":"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae","Type":"ContainerStarted","Data":"79a31cfaa05ad3b80231a6a877e248da49fc65efc7c6d967b5ed22be1634db02"}
Nov 03 23:39:36.365134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:36.364827    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:36.397074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:36.396985    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5" podStartSLOduration=2.396967352 podStartE2EDuration="2.396967352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:36.395723438 +0000 UTC m=+641.495323582" watchObservedRunningTime="2026-11-03 23:39:36.396967352 +0000 UTC m=+641.496567492"
Nov 03 23:39:38.381182 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:38.381150    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8" event={"ID":"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6","Type":"ContainerStarted","Data":"5ec0ece8f00b973b9a227b2e0efdb3eec53d5cea1c52d27c00804ed325135a30"}
Nov 03 23:39:38.404929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:38.404878    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8" podStartSLOduration=1.304107883 podStartE2EDuration="4.404843821s" totalImagesPullingTime="3.100735938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:38.402982378 +0000 UTC m=+643.502582523" watchObservedRunningTime="2026-11-03 23:39:38.404843821 +0000 UTC m=+643.504443964"
Nov 03 23:39:39.387957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:39.387920    5074 generic.go:356] "Generic (PLEG): container finished" podID="d33517df-f607-41a6-98df-ee77c5d4496c" containerID="be75f69877dfc339ba4684e43168eeafeefd647d446f4539535e2883dfebee84" exitCode=0
Nov 03 23:39:39.388381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:39.388007    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8" event={"ID":"d33517df-f607-41a6-98df-ee77c5d4496c","Type":"ContainerDied","Data":"be75f69877dfc339ba4684e43168eeafeefd647d446f4539535e2883dfebee84"}
Nov 03 23:39:39.389608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:39.389575    5074 generic.go:356] "Generic (PLEG): container finished" podID="3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" containerID="5ec0ece8f00b973b9a227b2e0efdb3eec53d5cea1c52d27c00804ed325135a30" exitCode=0
Nov 03 23:39:39.389781 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:39.389642    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8" event={"ID":"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6","Type":"ContainerDied","Data":"5ec0ece8f00b973b9a227b2e0efdb3eec53d5cea1c52d27c00804ed325135a30"}
Nov 03 23:39:40.551378 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.551345    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:40.564151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.563956    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca\") pod \"d33517df-f607-41a6-98df-ee77c5d4496c\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") "
Nov 03 23:39:40.564151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.564023    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw\" (UniqueName: \"kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw\") pod \"d33517df-f607-41a6-98df-ee77c5d4496c\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") "
Nov 03 23:39:40.564151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.564063    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca\") pod \"d33517df-f607-41a6-98df-ee77c5d4496c\" (UID: \"d33517df-f607-41a6-98df-ee77c5d4496c\") "
Nov 03 23:39:40.564495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.564352    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca" pod "d33517df-f607-41a6-98df-ee77c5d4496c" (UID: "d33517df-f607-41a6-98df-ee77c5d4496c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:40.566732 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.566706    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw" pod "d33517df-f607-41a6-98df-ee77c5d4496c" (UID: "d33517df-f607-41a6-98df-ee77c5d4496c"). InnerVolumeSpecName "kube-api-access-jnmqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:40.566944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.566918    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca" pod "d33517df-f607-41a6-98df-ee77c5d4496c" (UID: "d33517df-f607-41a6-98df-ee77c5d4496c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:40.585560 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.585538    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:40.664658 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.664577    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb\" (UniqueName: \"kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb\") pod \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") "
Nov 03 23:39:40.664833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.664680    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca\") pod \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") "
Nov 03 23:39:40.664833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.664732    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca\") pod \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\" (UID: \"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6\") "
Nov 03 23:39:40.665087 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.665070    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:40.665147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.665089    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmqw\" (UniqueName: \"kubernetes.io/projected/d33517df-f607-41a6-98df-ee77c5d4496c-kube-api-access-jnmqw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:40.665147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.665086    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca" pod "3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" (UID: "3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:40.665147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.665099    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33517df-f607-41a6-98df-ee77c5d4496c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:40.666926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.666807    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca" pod "3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" (UID: "3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:40.666926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.666887    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb" pod "3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" (UID: "3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6"). InnerVolumeSpecName "kube-api-access-tgxmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:40.765794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.765735    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgxmb\" (UniqueName: \"kubernetes.io/projected/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-kube-api-access-tgxmb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:40.765794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.765796    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:40.765982 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:40.765814    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:41.335369 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.335293    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:39:41.335369 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.335361    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:39:41.406802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.406773    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"
Nov 03 23:39:41.406964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.406780    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8" event={"ID":"3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6","Type":"ContainerDied","Data":"1652c12774a5bcdcfce84f32772d94f5731e7ed84a8da84da65ff25a46b4bdd6"}
Nov 03 23:39:41.406964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.406911    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1652c12774a5bcdcfce84f32772d94f5731e7ed84a8da84da65ff25a46b4bdd6"
Nov 03 23:39:41.409667 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.408957    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8" event={"ID":"d33517df-f607-41a6-98df-ee77c5d4496c","Type":"ContainerDied","Data":"118cddc4bc92780bf2fbb1f8e4e9940edefb54a2eaa581b57b819ec51cdd5ac8"}
Nov 03 23:39:41.409667 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.409005    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="118cddc4bc92780bf2fbb1f8e4e9940edefb54a2eaa581b57b819ec51cdd5ac8"
Nov 03 23:39:41.409667 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.409064    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"
Nov 03 23:39:41.432459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.432429    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"]
Nov 03 23:39:41.433189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.433165    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" containerName="revoke-keys"
Nov 03 23:39:41.433189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.433188    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" containerName="revoke-keys"
Nov 03 23:39:41.433428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.433198    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d33517df-f607-41a6-98df-ee77c5d4496c" containerName="revoke-keys"
Nov 03 23:39:41.433428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.433203    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33517df-f607-41a6-98df-ee77c5d4496c" containerName="revoke-keys"
Nov 03 23:39:41.433428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.433295    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" containerName="revoke-keys"
Nov 03 23:39:41.433428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.433303    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33517df-f607-41a6-98df-ee77c5d4496c" containerName="revoke-keys"
Nov 03 23:39:41.448102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.443553    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.448102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.447600    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"]
Nov 03 23:39:41.451025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.449851    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:39:41.451025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.450693    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:39:41.470883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.470854    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.471007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.470912    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.471007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.470973    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wbj\" (UniqueName: \"kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.571944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.571908    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.572428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.571964    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.572428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.571994    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wbj\" (UniqueName: \"kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.572618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.572598    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.574460 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.574435    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.584937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.584912    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wbj\" (UniqueName: \"kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj\") pod \"maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:41.766195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:41.766159    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:42.121798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:42.121686    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"]
Nov 03 23:39:42.150454 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:42.150422    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda246a7c9_1adc_4908_bc36_3871839522e7.slice/crio-5fea739319f6707b624d1af38f47a816fcf9d2c9488eb8c7c3e0fe4d66923009 WatchSource:0}: Error finding container 5fea739319f6707b624d1af38f47a816fcf9d2c9488eb8c7c3e0fe4d66923009: Status 404 returned error can't find the container with id 5fea739319f6707b624d1af38f47a816fcf9d2c9488eb8c7c3e0fe4d66923009
Nov 03 23:39:42.416032 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:42.415999    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg" event={"ID":"a246a7c9-1adc-4908-bc36-3871839522e7","Type":"ContainerStarted","Data":"5fea739319f6707b624d1af38f47a816fcf9d2c9488eb8c7c3e0fe4d66923009"}
Nov 03 23:39:43.423638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:43.423546    5074 generic.go:356] "Generic (PLEG): container finished" podID="a246a7c9-1adc-4908-bc36-3871839522e7" containerID="cd8b79781b2d42732bb4dc1dea4fcdd190440aa288fab9c08145bbf5db1f4546" exitCode=0
Nov 03 23:39:43.423638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:43.423603    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg" event={"ID":"a246a7c9-1adc-4908-bc36-3871839522e7","Type":"ContainerDied","Data":"cd8b79781b2d42732bb4dc1dea4fcdd190440aa288fab9c08145bbf5db1f4546"}
Nov 03 23:39:44.592974 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.592949    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:44.594707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.594688    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj\" (UniqueName: \"kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj\") pod \"a246a7c9-1adc-4908-bc36-3871839522e7\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") "
Nov 03 23:39:44.594833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.594814    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca\") pod \"a246a7c9-1adc-4908-bc36-3871839522e7\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") "
Nov 03 23:39:44.594899 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.594876    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca\") pod \"a246a7c9-1adc-4908-bc36-3871839522e7\" (UID: \"a246a7c9-1adc-4908-bc36-3871839522e7\") "
Nov 03 23:39:44.595146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.595118    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca" pod "a246a7c9-1adc-4908-bc36-3871839522e7" (UID: "a246a7c9-1adc-4908-bc36-3871839522e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:44.596714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.596692    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj" pod "a246a7c9-1adc-4908-bc36-3871839522e7" (UID: "a246a7c9-1adc-4908-bc36-3871839522e7"). InnerVolumeSpecName "kube-api-access-92wbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:44.596801 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.596783    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca" pod "a246a7c9-1adc-4908-bc36-3871839522e7" (UID: "a246a7c9-1adc-4908-bc36-3871839522e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:44.695664 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.695633    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:44.695664 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.695664    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92wbj\" (UniqueName: \"kubernetes.io/projected/a246a7c9-1adc-4908-bc36-3871839522e7-kube-api-access-92wbj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:44.695925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:44.695681    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a246a7c9-1adc-4908-bc36-3871839522e7-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:45.438330 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:45.438296    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"
Nov 03 23:39:45.438574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:45.438350    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg" event={"ID":"a246a7c9-1adc-4908-bc36-3871839522e7","Type":"ContainerDied","Data":"5fea739319f6707b624d1af38f47a816fcf9d2c9488eb8c7c3e0fe4d66923009"}
Nov 03 23:39:45.438574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:45.438412    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fea739319f6707b624d1af38f47a816fcf9d2c9488eb8c7c3e0fe4d66923009"
Nov 03 23:39:47.378629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:47.378594    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:39:48.983085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:48.983051    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"]
Nov 03 23:39:48.984992 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:48.984964    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-80942cbb-fkzx8"]
Nov 03 23:39:49.019991 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:49.019949    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6" path="/var/lib/kubelet/pods/3d10a2f8-a8d4-4b96-9c22-c80f32d90cd6/volumes"
Nov 03 23:39:50.042069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.042033    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"]
Nov 03 23:39:50.042484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.042471    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a246a7c9-1adc-4908-bc36-3871839522e7" containerName="revoke-keys"
Nov 03 23:39:50.042529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.042484    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a246a7c9-1adc-4908-bc36-3871839522e7" containerName="revoke-keys"
Nov 03 23:39:50.042600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.042591    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a246a7c9-1adc-4908-bc36-3871839522e7" containerName="revoke-keys"
Nov 03 23:39:50.046234 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.046205    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.065496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.062823    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:39:50.065663 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.065155    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"]
Nov 03 23:39:50.065951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.065695    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:39:50.149673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.149628    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.149861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.149690    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbw5c\" (UniqueName: \"kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.149861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.149716    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.251083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.251047    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.251270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.251126    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbw5c\" (UniqueName: \"kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.251270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.251162    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.251830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.251811    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.253493 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.253474    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.265097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.265076    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbw5c\" (UniqueName: \"kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c\") pod \"maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.360269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.360193    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:50.661115 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:50.661085    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"]
Nov 03 23:39:50.663923 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:50.663893    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96df9b51_c31e_4b53_b5d6_1b696f907121.slice/crio-3d2f271bfd2a85156f17bf0de77448d43ede9054ab1e0785d2c3eac144c04d07 WatchSource:0}: Error finding container 3d2f271bfd2a85156f17bf0de77448d43ede9054ab1e0785d2c3eac144c04d07: Status 404 returned error can't find the container with id 3d2f271bfd2a85156f17bf0de77448d43ede9054ab1e0785d2c3eac144c04d07
Nov 03 23:39:51.473517 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:51.473483    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9" event={"ID":"96df9b51-c31e-4b53-b5d6-1b696f907121","Type":"ContainerStarted","Data":"e90980d024912e180db8734d718b9bad0ba7fd19e05492f3c5a8eef4760e58b7"}
Nov 03 23:39:51.474037 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:51.473536    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9" event={"ID":"96df9b51-c31e-4b53-b5d6-1b696f907121","Type":"ContainerStarted","Data":"3d2f271bfd2a85156f17bf0de77448d43ede9054ab1e0785d2c3eac144c04d07"}
Nov 03 23:39:51.541365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:51.541329    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"]
Nov 03 23:39:51.545287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:51.545238    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9" podStartSLOduration=1.5452253809999998 podStartE2EDuration="1.545225381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:51.542555049 +0000 UTC m=+656.642155193" watchObservedRunningTime="2026-11-03 23:39:51.545225381 +0000 UTC m=+656.644825525"
Nov 03 23:39:51.546321 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:51.546298    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c2a2d28-5gcr8"]
Nov 03 23:39:52.480058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:52.480026    5074 generic.go:356] "Generic (PLEG): container finished" podID="96df9b51-c31e-4b53-b5d6-1b696f907121" containerID="e90980d024912e180db8734d718b9bad0ba7fd19e05492f3c5a8eef4760e58b7" exitCode=0
Nov 03 23:39:52.480435 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:52.480116    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9" event={"ID":"96df9b51-c31e-4b53-b5d6-1b696f907121","Type":"ContainerDied","Data":"e90980d024912e180db8734d718b9bad0ba7fd19e05492f3c5a8eef4760e58b7"}
Nov 03 23:39:53.016635 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.016594    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33517df-f607-41a6-98df-ee77c5d4496c" path="/var/lib/kubelet/pods/d33517df-f607-41a6-98df-ee77c5d4496c/volumes"
Nov 03 23:39:53.651871 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.651843    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:53.667142 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.667112    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"]
Nov 03 23:39:53.667569 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.667557    5074 cpu_manager.go:414] "containerMap: removing container" podUID="96df9b51-c31e-4b53-b5d6-1b696f907121" containerName="revoke-keys"
Nov 03 23:39:53.667617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.667570    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96df9b51-c31e-4b53-b5d6-1b696f907121" containerName="revoke-keys"
Nov 03 23:39:53.667662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.667653    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="96df9b51-c31e-4b53-b5d6-1b696f907121" containerName="revoke-keys"
Nov 03 23:39:53.670712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.670697    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.686366 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.686213    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"]
Nov 03 23:39:53.686526 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.686473    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c\" (UniqueName: \"kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c\") pod \"96df9b51-c31e-4b53-b5d6-1b696f907121\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") "
Nov 03 23:39:53.686526 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.686517    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca\") pod \"96df9b51-c31e-4b53-b5d6-1b696f907121\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") "
Nov 03 23:39:53.686715 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.686679    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca\") pod \"96df9b51-c31e-4b53-b5d6-1b696f907121\" (UID: \"96df9b51-c31e-4b53-b5d6-1b696f907121\") "
Nov 03 23:39:53.687883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.687838    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca" pod "96df9b51-c31e-4b53-b5d6-1b696f907121" (UID: "96df9b51-c31e-4b53-b5d6-1b696f907121"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:53.690539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.690518    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c" pod "96df9b51-c31e-4b53-b5d6-1b696f907121" (UID: "96df9b51-c31e-4b53-b5d6-1b696f907121"). InnerVolumeSpecName "kube-api-access-fbw5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:53.690663 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.690620    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca" pod "96df9b51-c31e-4b53-b5d6-1b696f907121" (UID: "96df9b51-c31e-4b53-b5d6-1b696f907121"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:53.788699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.788610    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.788891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.788773    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.788953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.788922    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cskd\" (UniqueName: \"kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.789049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.789026    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbw5c\" (UniqueName: \"kubernetes.io/projected/96df9b51-c31e-4b53-b5d6-1b696f907121-kube-api-access-fbw5c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:53.789049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.789050    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:53.789239 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.789068    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96df9b51-c31e-4b53-b5d6-1b696f907121-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:53.889656 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.889620    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cskd\" (UniqueName: \"kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.889818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.889672    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.889818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.889716    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.890403 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.890385    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.892008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.891989    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.919722 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.919690    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cskd\" (UniqueName: \"kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd\") pod \"maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:53.990267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:53.990226    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:54.286689 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:54.286654    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"]
Nov 03 23:39:54.289724 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:39:54.289698    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cb0da4_2119_4942_af43_0537d1e9f89d.slice/crio-81bbfc91415d5a9a06880b28fba9a474ea74116a9c8dcf3a83b665118a1fd134 WatchSource:0}: Error finding container 81bbfc91415d5a9a06880b28fba9a474ea74116a9c8dcf3a83b665118a1fd134: Status 404 returned error can't find the container with id 81bbfc91415d5a9a06880b28fba9a474ea74116a9c8dcf3a83b665118a1fd134
Nov 03 23:39:54.493286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:54.493250    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc" event={"ID":"f2cb0da4-2119-4942-af43-0537d1e9f89d","Type":"ContainerStarted","Data":"81bbfc91415d5a9a06880b28fba9a474ea74116a9c8dcf3a83b665118a1fd134"}
Nov 03 23:39:54.494699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:54.494676    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9" event={"ID":"96df9b51-c31e-4b53-b5d6-1b696f907121","Type":"ContainerDied","Data":"3d2f271bfd2a85156f17bf0de77448d43ede9054ab1e0785d2c3eac144c04d07"}
Nov 03 23:39:54.494837 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:54.494706    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"
Nov 03 23:39:54.494837 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:54.494714    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d2f271bfd2a85156f17bf0de77448d43ede9054ab1e0785d2c3eac144c04d07"
Nov 03 23:39:55.503156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:55.503054    5074 generic.go:356] "Generic (PLEG): container finished" podID="f2cb0da4-2119-4942-af43-0537d1e9f89d" containerID="79e1d0962fdb72db321505f87db0abcd03a8e244eaf41cfb81d3bf8c8f9b3e96" exitCode=0
Nov 03 23:39:55.503678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:55.503197    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc" event={"ID":"f2cb0da4-2119-4942-af43-0537d1e9f89d","Type":"ContainerDied","Data":"79e1d0962fdb72db321505f87db0abcd03a8e244eaf41cfb81d3bf8c8f9b3e96"}
Nov 03 23:39:56.633220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.633177    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"]
Nov 03 23:39:56.633671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.633230    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e9be6c8-6zhgg"]
Nov 03 23:39:56.669983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.669960    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:56.719530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.719490    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd\" (UniqueName: \"kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd\") pod \"f2cb0da4-2119-4942-af43-0537d1e9f89d\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") "
Nov 03 23:39:56.719889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.719863    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca\") pod \"f2cb0da4-2119-4942-af43-0537d1e9f89d\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") "
Nov 03 23:39:56.720062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.720043    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca\") pod \"f2cb0da4-2119-4942-af43-0537d1e9f89d\" (UID: \"f2cb0da4-2119-4942-af43-0537d1e9f89d\") "
Nov 03 23:39:56.720905 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.720883    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca" pod "f2cb0da4-2119-4942-af43-0537d1e9f89d" (UID: "f2cb0da4-2119-4942-af43-0537d1e9f89d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:56.724039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.723997    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd" pod "f2cb0da4-2119-4942-af43-0537d1e9f89d" (UID: "f2cb0da4-2119-4942-af43-0537d1e9f89d"). InnerVolumeSpecName "kube-api-access-8cskd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:56.728163 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.728132    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca" pod "f2cb0da4-2119-4942-af43-0537d1e9f89d" (UID: "f2cb0da4-2119-4942-af43-0537d1e9f89d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:56.821468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.821382    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:56.821468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.821419    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb0da4-2119-4942-af43-0537d1e9f89d-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:56.821468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:56.821434    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cskd\" (UniqueName: \"kubernetes.io/projected/f2cb0da4-2119-4942-af43-0537d1e9f89d-kube-api-access-8cskd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:57.017431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:57.017400    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a246a7c9-1adc-4908-bc36-3871839522e7" path="/var/lib/kubelet/pods/a246a7c9-1adc-4908-bc36-3871839522e7/volumes"
Nov 03 23:39:57.518727 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:57.518697    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"
Nov 03 23:39:57.518932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:57.518727    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc" event={"ID":"f2cb0da4-2119-4942-af43-0537d1e9f89d","Type":"ContainerDied","Data":"81bbfc91415d5a9a06880b28fba9a474ea74116a9c8dcf3a83b665118a1fd134"}
Nov 03 23:39:57.518932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:39:57.518797    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bbfc91415d5a9a06880b28fba9a474ea74116a9c8dcf3a83b665118a1fd134"
Nov 03 23:39:58.022172 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:39:58.022133    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:40:00.112985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:00.112952    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"]
Nov 03 23:40:00.121445 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:00.121416    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"]
Nov 03 23:40:05.113444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.113387    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl" podUID="7a69959b-0fc0-4d56-a300-9453c34644a3" containerName="payload-processing" containerID="cri-o://2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add" gracePeriod=25
Nov 03 23:40:05.122097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.122034    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl" podUID="9a451397-4b83-4a5e-ad65-f64f1071b288" containerName="payload-pre-processing" containerID="cri-o://7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed" gracePeriod=25
Nov 03 23:40:05.159937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.159904    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"]
Nov 03 23:40:05.206954 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.206927    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-044f4dcd-lp9p9"]
Nov 03 23:40:05.459681 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.456145    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:40:05.459681 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.458935    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:40:05.514503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.514467    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume\") pod \"7a69959b-0fc0-4d56-a300-9453c34644a3\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") "
Nov 03 23:40:05.514678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.514528    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm\" (UniqueName: \"kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm\") pod \"7a69959b-0fc0-4d56-a300-9453c34644a3\" (UID: \"7a69959b-0fc0-4d56-a300-9453c34644a3\") "
Nov 03 23:40:05.514678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.514584    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume\") pod \"9a451397-4b83-4a5e-ad65-f64f1071b288\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") "
Nov 03 23:40:05.514678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.514622    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8\" (UniqueName: \"kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8\") pod \"9a451397-4b83-4a5e-ad65-f64f1071b288\" (UID: \"9a451397-4b83-4a5e-ad65-f64f1071b288\") "
Nov 03 23:40:05.514998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.514976    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume" pod "7a69959b-0fc0-4d56-a300-9453c34644a3" (UID: "7a69959b-0fc0-4d56-a300-9453c34644a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:05.515068 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.515040    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume" pod "9a451397-4b83-4a5e-ad65-f64f1071b288" (UID: "9a451397-4b83-4a5e-ad65-f64f1071b288"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:05.516865 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.516826    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm" pod "7a69959b-0fc0-4d56-a300-9453c34644a3" (UID: "7a69959b-0fc0-4d56-a300-9453c34644a3"). InnerVolumeSpecName "kube-api-access-wmrtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:05.517151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.517128    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8" pod "9a451397-4b83-4a5e-ad65-f64f1071b288" (UID: "9a451397-4b83-4a5e-ad65-f64f1071b288"). InnerVolumeSpecName "kube-api-access-5g4l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:05.566913 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.566882    5074 generic.go:356] "Generic (PLEG): container finished" podID="7a69959b-0fc0-4d56-a300-9453c34644a3" containerID="2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add" exitCode=0
Nov 03 23:40:05.567097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.566924    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"
Nov 03 23:40:05.567097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.566969    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl" event={"ID":"7a69959b-0fc0-4d56-a300-9453c34644a3","Type":"ContainerDied","Data":"2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add"}
Nov 03 23:40:05.567097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.567039    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl" event={"ID":"7a69959b-0fc0-4d56-a300-9453c34644a3","Type":"ContainerDied","Data":"93827488360e9cb54e6425f38df38568e9c897214987e087614516647162146e"}
Nov 03 23:40:05.567097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.567069    5074 scope.go:122] "RemoveContainer" containerID="2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add"
Nov 03 23:40:05.568498 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.568460    5074 generic.go:356] "Generic (PLEG): container finished" podID="9a451397-4b83-4a5e-ad65-f64f1071b288" containerID="7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed" exitCode=0
Nov 03 23:40:05.568631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.568503    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"
Nov 03 23:40:05.568631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.568499    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl" event={"ID":"9a451397-4b83-4a5e-ad65-f64f1071b288","Type":"ContainerDied","Data":"7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed"}
Nov 03 23:40:05.568729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.568631    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl" event={"ID":"9a451397-4b83-4a5e-ad65-f64f1071b288","Type":"ContainerDied","Data":"1a870093e2ec792f8d6bc2e1735368c5c80ad93dcb44499cfda1a6de770bae25"}
Nov 03 23:40:05.579196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.578502    5074 scope.go:122] "RemoveContainer" containerID="2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add"
Nov 03 23:40:05.579196 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:40:05.578790    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add\": container with ID starting with 2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add not found: ID does not exist" containerID="2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add"
Nov 03 23:40:05.579196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.578823    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add"} err="failed to get container status \"2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add\": rpc error: code = NotFound desc = could not find container \"2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add\": container with ID starting with 2baab5824e2abaa185457e95cb502e9938e0cd196678c6a46d046522f3784add not found: ID does not exist"
Nov 03 23:40:05.579196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.578845    5074 scope.go:122] "RemoveContainer" containerID="7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed"
Nov 03 23:40:05.587775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.587748    5074 scope.go:122] "RemoveContainer" containerID="7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed"
Nov 03 23:40:05.588045 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:40:05.588021    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed\": container with ID starting with 7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed not found: ID does not exist" containerID="7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed"
Nov 03 23:40:05.588097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.588057    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed"} err="failed to get container status \"7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed\": rpc error: code = NotFound desc = could not find container \"7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed\": container with ID starting with 7283c83d165a938e05e0c804d167be3831c02c4bf6b005157b79e7f07bc904ed not found: ID does not exist"
Nov 03 23:40:05.610654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.610631    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"]
Nov 03 23:40:05.615787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.615768    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69959b-0fc0-4d56-a300-9453c34644a3-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:05.615878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.615790    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrtm\" (UniqueName: \"kubernetes.io/projected/7a69959b-0fc0-4d56-a300-9453c34644a3-kube-api-access-wmrtm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:05.615878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.615804    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a451397-4b83-4a5e-ad65-f64f1071b288-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:05.615878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.615816    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4l8\" (UniqueName: \"kubernetes.io/projected/9a451397-4b83-4a5e-ad65-f64f1071b288-kube-api-access-5g4l8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:05.617924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.617906    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-044f4dcd-689dffb5f8-gbbsl"]
Nov 03 23:40:05.665133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.665071    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"]
Nov 03 23:40:05.678426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:05.678398    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-044f4dcd-7b78df8dcf-tq2sl"]
Nov 03 23:40:06.065278 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:06.065245    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"]
Nov 03 23:40:07.017040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:07.016997    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a69959b-0fc0-4d56-a300-9453c34644a3" path="/var/lib/kubelet/pods/7a69959b-0fc0-4d56-a300-9453c34644a3/volumes"
Nov 03 23:40:07.017598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:07.017577    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96df9b51-c31e-4b53-b5d6-1b696f907121" path="/var/lib/kubelet/pods/96df9b51-c31e-4b53-b5d6-1b696f907121/volumes"
Nov 03 23:40:07.018001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:07.017980    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a451397-4b83-4a5e-ad65-f64f1071b288" path="/var/lib/kubelet/pods/9a451397-4b83-4a5e-ad65-f64f1071b288/volumes"
Nov 03 23:40:09.669694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.669658    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"]
Nov 03 23:40:09.670148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670132    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f2cb0da4-2119-4942-af43-0537d1e9f89d" containerName="revoke-keys"
Nov 03 23:40:09.670191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670149    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cb0da4-2119-4942-af43-0537d1e9f89d" containerName="revoke-keys"
Nov 03 23:40:09.670191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670178    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9a451397-4b83-4a5e-ad65-f64f1071b288" containerName="payload-pre-processing"
Nov 03 23:40:09.670191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670185    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a451397-4b83-4a5e-ad65-f64f1071b288" containerName="payload-pre-processing"
Nov 03 23:40:09.670289 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670195    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7a69959b-0fc0-4d56-a300-9453c34644a3" containerName="payload-processing"
Nov 03 23:40:09.670289 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670200    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a69959b-0fc0-4d56-a300-9453c34644a3" containerName="payload-processing"
Nov 03 23:40:09.670349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670293    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cb0da4-2119-4942-af43-0537d1e9f89d" containerName="revoke-keys"
Nov 03 23:40:09.670349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670306    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a451397-4b83-4a5e-ad65-f64f1071b288" containerName="payload-pre-processing"
Nov 03 23:40:09.670349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.670313    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a69959b-0fc0-4d56-a300-9453c34644a3" containerName="payload-processing"
Nov 03 23:40:09.673563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.673546    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.689083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.689064    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5738a9\""
Nov 03 23:40:09.690111 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.690092    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5738a9-dockercfg-jtwpj\""
Nov 03 23:40:09.722553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.722527    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"]
Nov 03 23:40:09.726006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.725988    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.754932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.754906    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jhn\" (UniqueName: \"kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn\") pod \"payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.755017 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.754942    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.768911 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.767884    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"]
Nov 03 23:40:09.771373 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.771348    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"]
Nov 03 23:40:09.856000 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.855962    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.856205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.856040    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jhn\" (UniqueName: \"kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn\") pod \"payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.856205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.856112    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.856330 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.856268    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmq5k\" (UniqueName: \"kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k\") pod \"payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.857382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.857147    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.905262 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.905229    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jhn\" (UniqueName: \"kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn\") pod \"payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:09.957095 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.957066    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmq5k\" (UniqueName: \"kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k\") pod \"payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.957275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.957197    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.957933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.957911    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.985012 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.984982    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmq5k\" (UniqueName: \"kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k\") pod \"payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:09.987624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:09.987605    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:10.038853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.038817    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:10.310657 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:40:10.310621    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d49ec7_a482_4f38_b933_b057ae376e4f.slice/crio-c0d0c1847f86462b62fd7d6d8304d184dfcc88303c5bfb53f486ffcd6301c926 WatchSource:0}: Error finding container c0d0c1847f86462b62fd7d6d8304d184dfcc88303c5bfb53f486ffcd6301c926: Status 404 returned error can't find the container with id c0d0c1847f86462b62fd7d6d8304d184dfcc88303c5bfb53f486ffcd6301c926
Nov 03 23:40:10.313030 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.312999    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"]
Nov 03 23:40:10.422791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.422726    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"]
Nov 03 23:40:10.423474 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:40:10.423447    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47312f3_f293_41d2_b581_a77372061952.slice/crio-1c5cc14a8787dd6ce27da99727849b19eeb78f07bc06d6105cfdfd9b88dc89f0 WatchSource:0}: Error finding container 1c5cc14a8787dd6ce27da99727849b19eeb78f07bc06d6105cfdfd9b88dc89f0: Status 404 returned error can't find the container with id 1c5cc14a8787dd6ce27da99727849b19eeb78f07bc06d6105cfdfd9b88dc89f0
Nov 03 23:40:10.598929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.598844    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4" event={"ID":"b2d49ec7-a482-4f38-b933-b057ae376e4f","Type":"ContainerStarted","Data":"b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd"}
Nov 03 23:40:10.598929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.598899    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4" event={"ID":"b2d49ec7-a482-4f38-b933-b057ae376e4f","Type":"ContainerStarted","Data":"c0d0c1847f86462b62fd7d6d8304d184dfcc88303c5bfb53f486ffcd6301c926"}
Nov 03 23:40:10.599140 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.598937    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:10.600508 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.600483    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4" event={"ID":"a47312f3-f293-41d2-b581-a77372061952","Type":"ContainerStarted","Data":"c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79"}
Nov 03 23:40:10.600615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.600527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4" event={"ID":"a47312f3-f293-41d2-b581-a77372061952","Type":"ContainerStarted","Data":"1c5cc14a8787dd6ce27da99727849b19eeb78f07bc06d6105cfdfd9b88dc89f0"}
Nov 03 23:40:10.600615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.600573    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:10.662883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.662834    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4" podStartSLOduration=1.662820552 podStartE2EDuration="1.662820552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:10.6590687 +0000 UTC m=+675.758668843" watchObservedRunningTime="2026-11-03 23:40:10.662820552 +0000 UTC m=+675.762420695"
Nov 03 23:40:10.720209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:10.720153    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4" podStartSLOduration=1.720139884 podStartE2EDuration="1.720139884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:10.716970697 +0000 UTC m=+675.816570841" watchObservedRunningTime="2026-11-03 23:40:10.720139884 +0000 UTC m=+675.819740031"
Nov 03 23:40:11.066300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.066220    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5" podUID="cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" containerName="payload-pre-processing" containerID="cri-o://03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603" gracePeriod=25
Nov 03 23:40:11.343879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.343851    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:40:11.478589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.478561    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume\") pod \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") "
Nov 03 23:40:11.478780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.478621    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm\" (UniqueName: \"kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm\") pod \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\" (UID: \"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae\") "
Nov 03 23:40:11.479030 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.479004    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume" pod "cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" (UID: "cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:11.480885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.480857    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm" pod "cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" (UID: "cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae"). InnerVolumeSpecName "kube-api-access-zm5hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:11.580311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.580231    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:11.580311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.580262    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5hm\" (UniqueName: \"kubernetes.io/projected/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae-kube-api-access-zm5hm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:11.606888 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.606856    5074 generic.go:356] "Generic (PLEG): container finished" podID="cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" containerID="03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603" exitCode=0
Nov 03 23:40:11.607035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.606901    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"
Nov 03 23:40:11.607035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.606940    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5" event={"ID":"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae","Type":"ContainerDied","Data":"03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603"}
Nov 03 23:40:11.607035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.606989    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5" event={"ID":"cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae","Type":"ContainerDied","Data":"79a31cfaa05ad3b80231a6a877e248da49fc65efc7c6d967b5ed22be1634db02"}
Nov 03 23:40:11.607035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.607018    5074 scope.go:122] "RemoveContainer" containerID="03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603"
Nov 03 23:40:11.620551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.620531    5074 scope.go:122] "RemoveContainer" containerID="03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603"
Nov 03 23:40:11.620926 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:40:11.620898    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603\": container with ID starting with 03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603 not found: ID does not exist" containerID="03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603"
Nov 03 23:40:11.621026 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.620939    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603"} err="failed to get container status \"03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603\": rpc error: code = NotFound desc = could not find container \"03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603\": container with ID starting with 03937d2d1623212d45539379768fc0d746fe5eef40ea0909c02de0378aeaf603 not found: ID does not exist"
Nov 03 23:40:11.661469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.661434    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"]
Nov 03 23:40:11.668569 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:11.668498    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d3157179-559bb949dc-g4bc5"]
Nov 03 23:40:13.016181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:13.016146    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" path="/var/lib/kubelet/pods/cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae/volumes"
Nov 03 23:40:13.127146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:13.127113    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"]
Nov 03 23:40:13.161501 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:13.161470    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d3157179-vmkwc"]
Nov 03 23:40:15.018399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:15.018368    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2cb0da4-2119-4942-af43-0537d1e9f89d" path="/var/lib/kubelet/pods/f2cb0da4-2119-4942-af43-0537d1e9f89d/volumes"
Nov 03 23:40:20.314539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.314506    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"]
Nov 03 23:40:20.314951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.314936    5074 cpu_manager.go:414] "containerMap: removing container" podUID="cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" containerName="payload-pre-processing"
Nov 03 23:40:20.314996 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.314952    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" containerName="payload-pre-processing"
Nov 03 23:40:20.315059 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.315050    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf518a4a-b7ad-4683-b1b7-f8f2c25f6bae" containerName="payload-pre-processing"
Nov 03 23:40:20.319474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.319457    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.339465 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.339439    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c60893-dockercfg-r2jwz\""
Nov 03 23:40:20.339607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.339527    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c60893\""
Nov 03 23:40:20.340626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.340601    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"]
Nov 03 23:40:20.472985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.472945    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.473167 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.473145    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8cd\" (UniqueName: \"kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd\") pod \"payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.574644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.574546    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.574834 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.574732    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8cd\" (UniqueName: \"kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd\") pod \"payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.575310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.575288    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.591318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.591288    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8cd\" (UniqueName: \"kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd\") pod \"payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.633195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.633147    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:20.930053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:20.930024    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"]
Nov 03 23:40:20.932049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:40:20.932020    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bea0eb_57d4_401c_abe4_888841ad40ae.slice/crio-a3db5f98558556b1e92dcd76a6e77f5994daea84b0d8876644d52b2881fdc446 WatchSource:0}: Error finding container a3db5f98558556b1e92dcd76a6e77f5994daea84b0d8876644d52b2881fdc446: Status 404 returned error can't find the container with id a3db5f98558556b1e92dcd76a6e77f5994daea84b0d8876644d52b2881fdc446
Nov 03 23:40:21.610514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:21.610484    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 03 23:40:21.610977 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:21.610620    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 03 23:40:21.678541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:21.678505    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw" event={"ID":"15bea0eb-57d4-401c-abe4-888841ad40ae","Type":"ContainerStarted","Data":"71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6"}
Nov 03 23:40:21.678781 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:21.678592    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw" event={"ID":"15bea0eb-57d4-401c-abe4-888841ad40ae","Type":"ContainerStarted","Data":"a3db5f98558556b1e92dcd76a6e77f5994daea84b0d8876644d52b2881fdc446"}
Nov 03 23:40:21.678867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:21.678798    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:28.022229 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:40:28.022189    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:40:32.202506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.202437    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw" podStartSLOduration=12.202415632 podStartE2EDuration="12.202415632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:21.726533208 +0000 UTC m=+686.826133353" watchObservedRunningTime="2026-11-03 23:40:32.202415632 +0000 UTC m=+697.302015780"
Nov 03 23:40:32.205260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.205226    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"]
Nov 03 23:40:32.209709 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.209684    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.218609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.218584    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5cb349\""
Nov 03 23:40:32.225940 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.221020    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5cb349-dockercfg-q2wjv\""
Nov 03 23:40:32.225940 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.222851    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"]
Nov 03 23:40:32.249217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.249185    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"]
Nov 03 23:40:32.276105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.276072    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.278304 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.278278    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"]
Nov 03 23:40:32.285872 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.285851    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cdp\" (UniqueName: \"kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp\") pod \"payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.286005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.285936    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.387455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.387418    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfs6\" (UniqueName: \"kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6\") pod \"payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.387647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.387479    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.387647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.387550    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.387647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.387626    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cdp\" (UniqueName: \"kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp\") pod \"payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.388363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.388337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.401155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.401127    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cdp\" (UniqueName: \"kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp\") pod \"payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.488439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.488351    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfs6\" (UniqueName: \"kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6\") pod \"payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.488439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.488417    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.489370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.489344    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.504311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.504281    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfs6\" (UniqueName: \"kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6\") pod \"payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.529012 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.528982    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:32.594227 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.593989    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:32.686231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.686205    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 03 23:40:32.823497 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.823464    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"]
Nov 03 23:40:32.824430 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:40:32.824400    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27b72ea_653a_47e3_9330_312df17c16cb.slice/crio-9c2798abee896b4aaec02f16c1e6e15aa627c9e5c7eef8451c62529220d699cb WatchSource:0}: Error finding container 9c2798abee896b4aaec02f16c1e6e15aa627c9e5c7eef8451c62529220d699cb: Status 404 returned error can't find the container with id 9c2798abee896b4aaec02f16c1e6e15aa627c9e5c7eef8451c62529220d699cb
Nov 03 23:40:32.891486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:32.891466    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"]
Nov 03 23:40:32.892026 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:40:32.892004    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89909b81_d1a4_4134_9692_9544cba7903b.slice/crio-530781b24973cdf0380e8f60866a9efa808596e15baf115655dce76fe31ba158 WatchSource:0}: Error finding container 530781b24973cdf0380e8f60866a9efa808596e15baf115655dce76fe31ba158: Status 404 returned error can't find the container with id 530781b24973cdf0380e8f60866a9efa808596e15baf115655dce76fe31ba158
Nov 03 23:40:33.760112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.760075    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l" event={"ID":"89909b81-d1a4-4134-9692-9544cba7903b","Type":"ContainerStarted","Data":"de875d08255e379f7e1bd5214cce074e104d7e074563c1c0a6997a7c51a4d96c"}
Nov 03 23:40:33.760556 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.760134    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l" event={"ID":"89909b81-d1a4-4134-9692-9544cba7903b","Type":"ContainerStarted","Data":"530781b24973cdf0380e8f60866a9efa808596e15baf115655dce76fe31ba158"}
Nov 03 23:40:33.760556 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.760166    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:33.762098 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.762066    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7" event={"ID":"a27b72ea-653a-47e3-9330-312df17c16cb","Type":"ContainerStarted","Data":"8e0efaf924feaf67f7a049b4faf8105462bd94d96bfc62e2d7549f1063671a3f"}
Nov 03 23:40:33.762255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.762119    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7" event={"ID":"a27b72ea-653a-47e3-9330-312df17c16cb","Type":"ContainerStarted","Data":"9c2798abee896b4aaec02f16c1e6e15aa627c9e5c7eef8451c62529220d699cb"}
Nov 03 23:40:33.762255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.762147    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:33.796596 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.796552    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l" podStartSLOduration=1.796539663 podStartE2EDuration="1.796539663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:33.794334121 +0000 UTC m=+698.893934280" watchObservedRunningTime="2026-11-03 23:40:33.796539663 +0000 UTC m=+698.896139806"
Nov 03 23:40:33.825211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:33.825156    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7" podStartSLOduration=1.825139021 podStartE2EDuration="1.825139021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:33.824533642 +0000 UTC m=+698.924133789" watchObservedRunningTime="2026-11-03 23:40:33.825139021 +0000 UTC m=+698.924739166"
Nov 03 23:40:44.772866 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:44.772827    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 03 23:40:44.775452 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:44.772896    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 03 23:40:52.136461 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.136427    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"]
Nov 03 23:40:52.140520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.140495    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.168241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.168210    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"kube-root-ca.crt\""
Nov 03 23:40:52.168390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.168361    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"openshift-service-ca.crt\""
Nov 03 23:40:52.171816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.171792    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"]
Nov 03 23:40:52.174643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.174613    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"default-dockercfg-926zp\""
Nov 03 23:40:52.174954 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.174936    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"premium-w0-f6f292-kserve-self-signed-certs\""
Nov 03 23:40:52.308420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.308384    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9fq\" (UniqueName: \"kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.308631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.308451    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.308631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.308556    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.308772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.308671    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.308849 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.308794    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.308912 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.308858    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.409912    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.409949    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.409997    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410284 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.410060    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410284 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.410113    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410284 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.410155    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9fq\" (UniqueName: \"kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.410407    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410680 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.410657    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.410927 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.410910    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.413168 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.413141    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.413293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.413182    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.430642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.430619    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9fq\" (UniqueName: \"kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq\") pod \"premium-w0-f6f292-kserve-9b758cb99-gml5j\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") " pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.473057 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.473024    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:52.806480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.806446    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"]
Nov 03 23:40:52.807769 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:40:52.807727    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod670271d0_1d7e_4041_97dd_82d07d99e5c0.slice/crio-bd04b4e759fb704bd32aeca268372634feb7d7c402349e6cbecf77fca69d6810 WatchSource:0}: Error finding container bd04b4e759fb704bd32aeca268372634feb7d7c402349e6cbecf77fca69d6810: Status 404 returned error can't find the container with id bd04b4e759fb704bd32aeca268372634feb7d7c402349e6cbecf77fca69d6810
Nov 03 23:40:52.884211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:52.884174    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" event={"ID":"670271d0-1d7e-4041-97dd-82d07d99e5c0","Type":"ContainerStarted","Data":"bd04b4e759fb704bd32aeca268372634feb7d7c402349e6cbecf77fca69d6810"}
Nov 03 23:40:54.898621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:54.898582    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" event={"ID":"670271d0-1d7e-4041-97dd-82d07d99e5c0","Type":"ContainerStarted","Data":"d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f"}
Nov 03 23:40:54.920527 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:54.920468    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podStartSLOduration=1.488516494 podStartE2EDuration="2.920456047s" totalImagesPullingTime="1.431939553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:54.919721797 +0000 UTC m=+720.019321956" watchObservedRunningTime="2026-11-03 23:40:54.920456047 +0000 UTC m=+720.020056193"
Nov 03 23:40:55.473716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:40:55.473674    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:40:58.022147 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:40:58.022112    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:41:00.553547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.553511    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"]
Nov 03 23:41:00.559176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.559151    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.570529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.566578    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"openshift-service-ca.crt\""
Nov 03 23:41:00.570529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.566651    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"default-dockercfg-5kdv6\""
Nov 03 23:41:00.570529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.567563    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"premium-w2-5738a9-kserve-self-signed-certs\""
Nov 03 23:41:00.570529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.567591    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"kube-root-ca.crt\""
Nov 03 23:41:00.570529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.568034    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"]
Nov 03 23:41:00.602760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.599186    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.602760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.599233    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.602760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.599429    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.602760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.599478    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.602760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.599512    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.602760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.599533    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7mx\" (UniqueName: \"kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.701535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.700731    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.701535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.700830    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.701535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.700866    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.701535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.700893    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7mx\" (UniqueName: \"kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.701535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.700939    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.701535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.700963    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.706806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.702365    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.707024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.706996    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.707271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.707247    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.721051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.720939    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.721051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.720977    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.725510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.725462    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7mx\" (UniqueName: \"kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx\") pod \"premium-w2-5738a9-kserve-6f6f568887-8vdq4\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") " pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:00.878371 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:00.878288    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:01.163783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:01.163751    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"]
Nov 03 23:41:01.165328 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:01.165291    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73141bd1_58b2_4f0c_9fda_b395ca8c81a6.slice/crio-99c2415df60bc66b2b2c674a4381b62e4780647afe56c7f0d87e2c065ad4c9f5 WatchSource:0}: Error finding container 99c2415df60bc66b2b2c674a4381b62e4780647afe56c7f0d87e2c065ad4c9f5: Status 404 returned error can't find the container with id 99c2415df60bc66b2b2c674a4381b62e4780647afe56c7f0d87e2c065ad4c9f5
Nov 03 23:41:01.942781 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:01.942718    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" event={"ID":"73141bd1-58b2-4f0c-9fda-b395ca8c81a6","Type":"ContainerStarted","Data":"99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5"}
Nov 03 23:41:01.943204 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:01.942793    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" event={"ID":"73141bd1-58b2-4f0c-9fda-b395ca8c81a6","Type":"ContainerStarted","Data":"99c2415df60bc66b2b2c674a4381b62e4780647afe56c7f0d87e2c065ad4c9f5"}
Nov 03 23:41:01.965576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:01.965520    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podStartSLOduration=1.965503822 podStartE2EDuration="1.965503822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:01.96333328 +0000 UTC m=+727.062933425" watchObservedRunningTime="2026-11-03 23:41:01.965503822 +0000 UTC m=+727.065103967"
Nov 03 23:41:02.474250 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:02.474214    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:41:02.492187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:02.492160    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:41:02.879136 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:02.879065    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:02.960113 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:02.960087    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:41:10.879065 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:10.879027    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:10.895101 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:10.894963    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:11.018172 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:11.018143    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:41:15.328710 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.328659    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"]
Nov 03 23:41:15.336348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.336326    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.342522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.342494    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"openshift-service-ca.crt\""
Nov 03 23:41:15.345270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.345242    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"]
Nov 03 23:41:15.352085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.352056    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"premium-w1-5cb349-kserve-self-signed-certs\""
Nov 03 23:41:15.352279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.352065    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"kube-root-ca.crt\""
Nov 03 23:41:15.352279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.352238    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"default-dockercfg-6tjp2\""
Nov 03 23:41:15.454769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.453733    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.454769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.453865    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.454769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.453967    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.454769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.454141    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rwx\" (UniqueName: \"kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.454769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.454172    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.454769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.454215    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555098 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555065    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555098 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555106    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555257    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555358    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rwx\" (UniqueName: \"kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555390    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555435    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555640 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555615    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555888 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555866    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.555997 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.555907    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.557628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.557599    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.557922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.557905    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.568929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.568637    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rwx\" (UniqueName: \"kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx\") pod \"premium-w1-5cb349-kserve-6f787c957-v5czs\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") " pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.665391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.665312    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:15.950503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:15.950473    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"]
Nov 03 23:41:15.951400 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:15.951369    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f2a4304_fe1d_450b_88b6_f3547bc1c87b.slice/crio-52fed4d5d777030cac964e0d2c062500afd2db53c642fd0683fb7036e1a7e176 WatchSource:0}: Error finding container 52fed4d5d777030cac964e0d2c062500afd2db53c642fd0683fb7036e1a7e176: Status 404 returned error can't find the container with id 52fed4d5d777030cac964e0d2c062500afd2db53c642fd0683fb7036e1a7e176
Nov 03 23:41:16.034384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:16.034351    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" event={"ID":"4f2a4304-fe1d-450b-88b6-f3547bc1c87b","Type":"ContainerStarted","Data":"653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba"}
Nov 03 23:41:16.034504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:16.034395    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" event={"ID":"4f2a4304-fe1d-450b-88b6-f3547bc1c87b","Type":"ContainerStarted","Data":"52fed4d5d777030cac964e0d2c062500afd2db53c642fd0683fb7036e1a7e176"}
Nov 03 23:41:16.056046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:16.055997    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podStartSLOduration=1.055983757 podStartE2EDuration="1.055983757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:16.055046607 +0000 UTC m=+741.154646754" watchObservedRunningTime="2026-11-03 23:41:16.055983757 +0000 UTC m=+741.155583898"
Nov 03 23:41:16.666008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:16.665972    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:25.666177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:25.666135    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:25.680763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:25.680716    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:26.117774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:26.117725    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:41:28.021476 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:41:28.021439    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:41:28.607712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.607677    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"]
Nov 03 23:41:28.612116 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.612094    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.619346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.619314    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:41:28.619494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.619374    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:41:28.620206 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.620183    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"]
Nov 03 23:41:28.694638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.694608    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7s86\" (UniqueName: \"kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.694844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.694647    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.694844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.694672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.795471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.795436    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7s86\" (UniqueName: \"kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.795644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.795480    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.795644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.795526    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.796246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.796222    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.797805 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.797785    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.808800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.808775    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7s86\" (UniqueName: \"kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86\") pod \"maas-api-revoke-sub-50c0bc15-ng7lh\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:28.929562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:28.929475    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:29.214668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:29.214635    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"]
Nov 03 23:41:29.216263 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:29.216237    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd4964b_7ec4_4a92_a907_e6f3d1fddb9d.slice/crio-5dec1153bf40bc194d3aeee33b8310191de0f9fbc702420fe3afd3739e908154 WatchSource:0}: Error finding container 5dec1153bf40bc194d3aeee33b8310191de0f9fbc702420fe3afd3739e908154: Status 404 returned error can't find the container with id 5dec1153bf40bc194d3aeee33b8310191de0f9fbc702420fe3afd3739e908154
Nov 03 23:41:30.132906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:30.132868    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh" event={"ID":"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d","Type":"ContainerStarted","Data":"4d7159e03f549c19cf1158a2669737393e8648ed4f07d2028be386045d2a1109"}
Nov 03 23:41:30.133092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:30.132968    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh" event={"ID":"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d","Type":"ContainerStarted","Data":"5dec1153bf40bc194d3aeee33b8310191de0f9fbc702420fe3afd3739e908154"}
Nov 03 23:41:30.156636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:30.156583    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh" podStartSLOduration=2.156570184 podStartE2EDuration="2.156570184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:30.155501087 +0000 UTC m=+755.255101232" watchObservedRunningTime="2026-11-03 23:41:30.156570184 +0000 UTC m=+755.256170328"
Nov 03 23:41:31.140717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:31.140683    5074 generic.go:356] "Generic (PLEG): container finished" podID="0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" containerID="4d7159e03f549c19cf1158a2669737393e8648ed4f07d2028be386045d2a1109" exitCode=0
Nov 03 23:41:31.141196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:31.140788    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh" event={"ID":"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d","Type":"ContainerDied","Data":"4d7159e03f549c19cf1158a2669737393e8648ed4f07d2028be386045d2a1109"}
Nov 03 23:41:32.306454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.306429    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:32.444182 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.444144    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca\") pod \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") "
Nov 03 23:41:32.444351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.444285    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86\" (UniqueName: \"kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86\") pod \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") "
Nov 03 23:41:32.444351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.444320    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca\") pod \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\" (UID: \"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d\") "
Nov 03 23:41:32.444692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.444668    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca" pod "0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" (UID: "0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:41:32.446408 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.446385    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca" pod "0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" (UID: "0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:41:32.446408 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.446391    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86" pod "0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" (UID: "0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d"). InnerVolumeSpecName "kube-api-access-v7s86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:41:32.546214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.546175    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7s86\" (UniqueName: \"kubernetes.io/projected/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-kube-api-access-v7s86\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:32.546214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.546208    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:32.546414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:32.546227    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:33.155602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:33.155571    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"
Nov 03 23:41:33.155602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:33.155579    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh" event={"ID":"0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d","Type":"ContainerDied","Data":"5dec1153bf40bc194d3aeee33b8310191de0f9fbc702420fe3afd3739e908154"}
Nov 03 23:41:33.155827 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:33.155626    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dec1153bf40bc194d3aeee33b8310191de0f9fbc702420fe3afd3739e908154"
Nov 03 23:41:37.485355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.485269    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"]
Nov 03 23:41:37.485845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.485827    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" containerName="revoke-keys"
Nov 03 23:41:37.485951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.485846    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" containerName="revoke-keys"
Nov 03 23:41:37.486005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.485993    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" containerName="revoke-keys"
Nov 03 23:41:37.489497 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.489477    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493461 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493438    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493474    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493514    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493640    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493710 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493688    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493713    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfdb\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493871 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493850    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.493924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493883    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.494012 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.493991    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.497930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.497903    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-80f8ba-openshift-default-dockercfg-rx8cf\""
Nov 03 23:41:37.502193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.502170    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"]
Nov 03 23:41:37.595095 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595063    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595108    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595125    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595232    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfdb\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595284    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595412    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595451    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595493    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595535    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595495    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595610    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.595821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.595685    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.596055 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.596036    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.597641 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.597617    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.598006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.597975    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.608717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.608691    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.608926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.608907    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfdb\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:37.803419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:37.803333    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:38.096509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.096483    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"]
Nov 03 23:41:38.097974 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:38.097947    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013b9e70_4fe3_478f_b31b_0138f2d2dd7b.slice/crio-7bff755ac0bfffbdade387a1c25dfb0291331f097f14edc59979f18cf50590c2 WatchSource:0}: Error finding container 7bff755ac0bfffbdade387a1c25dfb0291331f097f14edc59979f18cf50590c2: Status 404 returned error can't find the container with id 7bff755ac0bfffbdade387a1c25dfb0291331f097f14edc59979f18cf50590c2
Nov 03 23:41:38.100905 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.100865    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:41:38.101001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.100957    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:41:38.101042 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.101004    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:41:38.186222 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.186183    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" event={"ID":"013b9e70-4fe3-478f-b31b-0138f2d2dd7b","Type":"ContainerStarted","Data":"1be59122d087c7c323b98610e43e59337a27cbf773a2fc20169d9a4aac19fa3d"}
Nov 03 23:41:38.186387 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.186262    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" event={"ID":"013b9e70-4fe3-478f-b31b-0138f2d2dd7b","Type":"ContainerStarted","Data":"7bff755ac0bfffbdade387a1c25dfb0291331f097f14edc59979f18cf50590c2"}
Nov 03 23:41:38.213865 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.213816    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podStartSLOduration=1.213801778 podStartE2EDuration="1.213801778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:38.210404571 +0000 UTC m=+763.310004719" watchObservedRunningTime="2026-11-03 23:41:38.213801778 +0000 UTC m=+763.313401922"
Nov 03 23:41:38.333302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.333269    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"]
Nov 03 23:41:38.337291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.337268    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.347339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.347318    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"]
Nov 03 23:41:38.347910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.347873    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:41:38.348009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.347940    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:41:38.405371 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.405340    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.405545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.405396    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjb85\" (UniqueName: \"kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.405632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.405539    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.507244 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.507174    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.507719 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.507279    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjb85\" (UniqueName: \"kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.507719 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.507371    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.508139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.508108    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.509927 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.509901    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.521891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.521865    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjb85\" (UniqueName: \"kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85\") pod \"maas-api-revoke-sub-1795fca2-8t84l\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.656156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.656079    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:38.813762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.811084    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:38.968542 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:38.968517    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"]
Nov 03 23:41:38.970453 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:38.970422    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32080b3_f130_47b7_ab1c_f3a20504de7c.slice/crio-3f3b659e9b3979e170cb142bb28235d2561b34529169dcf6147727d9239142f2 WatchSource:0}: Error finding container 3f3b659e9b3979e170cb142bb28235d2561b34529169dcf6147727d9239142f2: Status 404 returned error can't find the container with id 3f3b659e9b3979e170cb142bb28235d2561b34529169dcf6147727d9239142f2
Nov 03 23:41:39.130654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.130293    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"]
Nov 03 23:41:39.136374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.136351    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.140117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.140090    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"]
Nov 03 23:41:39.193726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.193696    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l" event={"ID":"c32080b3-f130-47b7-ab1c-f3a20504de7c","Type":"ContainerStarted","Data":"3f3b659e9b3979e170cb142bb28235d2561b34529169dcf6147727d9239142f2"}
Nov 03 23:41:39.239814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.239257    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.239814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.239364    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2sl\" (UniqueName: \"kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.239814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.239474    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.341027 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.340992    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.341208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.341120    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.341208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.341172    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2sl\" (UniqueName: \"kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.341868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.341843    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.343440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.343418    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.386386 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.386357    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2sl\" (UniqueName: \"kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl\") pod \"maas-api-revoke-sub-85eb01ec-g9g9v\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.454334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.454298    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:39.771684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.771657    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"]
Nov 03 23:41:39.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.807234    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 23:41:39.821761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:39.807286    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 23:41:40.218458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:40.217954    5074 generic.go:356] "Generic (PLEG): container finished" podID="c32080b3-f130-47b7-ab1c-f3a20504de7c" containerID="706a63656cf93f111e7b36766b3b4ffc6b5f3fe0852edba44108d8eec49b2747" exitCode=0
Nov 03 23:41:40.218458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:40.218096    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l" event={"ID":"c32080b3-f130-47b7-ab1c-f3a20504de7c","Type":"ContainerDied","Data":"706a63656cf93f111e7b36766b3b4ffc6b5f3fe0852edba44108d8eec49b2747"}
Nov 03 23:41:40.221118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:40.221054    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v" event={"ID":"110009d6-c3ce-4f74-9c0b-fcbddfc390f4","Type":"ContainerStarted","Data":"5a84aee70511a4f08fe117e612fea4bfa80c49a6cfcd6910984160b6f1f87f06"}
Nov 03 23:41:40.818762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:40.811167    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 23:41:40.818762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:40.811226    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 23:41:41.240605 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.240545    5074 generic.go:356] "Generic (PLEG): container finished" podID="110009d6-c3ce-4f74-9c0b-fcbddfc390f4" containerID="2915856aec18487baaef69742df281f5d42fe238e0121748c5308499487185bd" exitCode=0
Nov 03 23:41:41.240829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.240633    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v" event={"ID":"110009d6-c3ce-4f74-9c0b-fcbddfc390f4","Type":"ContainerDied","Data":"2915856aec18487baaef69742df281f5d42fe238e0121748c5308499487185bd"}
Nov 03 23:41:41.470667 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.470640    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:41.567557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.567471    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca\") pod \"c32080b3-f130-47b7-ab1c-f3a20504de7c\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") "
Nov 03 23:41:41.567557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.567541    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca\") pod \"c32080b3-f130-47b7-ab1c-f3a20504de7c\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") "
Nov 03 23:41:41.567768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.567603    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85\" (UniqueName: \"kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85\") pod \"c32080b3-f130-47b7-ab1c-f3a20504de7c\" (UID: \"c32080b3-f130-47b7-ab1c-f3a20504de7c\") "
Nov 03 23:41:41.567946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.567922    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca" pod "c32080b3-f130-47b7-ab1c-f3a20504de7c" (UID: "c32080b3-f130-47b7-ab1c-f3a20504de7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:41:41.569722 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.569698    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca" pod "c32080b3-f130-47b7-ab1c-f3a20504de7c" (UID: "c32080b3-f130-47b7-ab1c-f3a20504de7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:41:41.569859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.569779    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85" pod "c32080b3-f130-47b7-ab1c-f3a20504de7c" (UID: "c32080b3-f130-47b7-ab1c-f3a20504de7c"). InnerVolumeSpecName "kube-api-access-sjb85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:41:41.668689 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.668653    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:41.668689 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.668689    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32080b3-f130-47b7-ab1c-f3a20504de7c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:41.668937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.668709    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjb85\" (UniqueName: \"kubernetes.io/projected/c32080b3-f130-47b7-ab1c-f3a20504de7c-kube-api-access-sjb85\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:41.801765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.799604    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"]
Nov 03 23:41:41.815763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.807140    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 23:41:41.815763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.807201    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 23:41:41.815763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.808278    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c32080b3-f130-47b7-ab1c-f3a20504de7c" containerName="revoke-keys"
Nov 03 23:41:41.815763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.808299    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32080b3-f130-47b7-ab1c-f3a20504de7c" containerName="revoke-keys"
Nov 03 23:41:41.815763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.808449    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32080b3-f130-47b7-ab1c-f3a20504de7c" containerName="revoke-keys"
Nov 03 23:41:41.826531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.825015    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.847328 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.847298    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"]
Nov 03 23:41:41.871009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.870972    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.871232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.871017    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dfm\" (UniqueName: \"kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.871232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.871146    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.972367 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.972329    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.972662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.972387    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dfm\" (UniqueName: \"kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.972662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.972473    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.973014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.972993    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.974789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.974757    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:41.985485 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:41.985448    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dfm\" (UniqueName: \"kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm\") pod \"maas-api-revoke-sub-881dcbc4-nhhwc\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:42.147708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.147604    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:42.264235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.264208    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"
Nov 03 23:41:42.264632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.264608    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l" event={"ID":"c32080b3-f130-47b7-ab1c-f3a20504de7c","Type":"ContainerDied","Data":"3f3b659e9b3979e170cb142bb28235d2561b34529169dcf6147727d9239142f2"}
Nov 03 23:41:42.264714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.264662    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3b659e9b3979e170cb142bb28235d2561b34529169dcf6147727d9239142f2"
Nov 03 23:41:42.496410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.496385    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:42.528822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.528793    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"]
Nov 03 23:41:42.529212 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:42.529188    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177ba727_56bb_496a_82a7_4bafe41aab8a.slice/crio-510087b9a17ccc3296463405f21e477779c10ea27a949132a730a17d07168e1d WatchSource:0}: Error finding container 510087b9a17ccc3296463405f21e477779c10ea27a949132a730a17d07168e1d: Status 404 returned error can't find the container with id 510087b9a17ccc3296463405f21e477779c10ea27a949132a730a17d07168e1d
Nov 03 23:41:42.581828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.581791    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca\") pod \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") "
Nov 03 23:41:42.582036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.582002    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca\") pod \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") "
Nov 03 23:41:42.582144 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.582114    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl\" (UniqueName: \"kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl\") pod \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\" (UID: \"110009d6-c3ce-4f74-9c0b-fcbddfc390f4\") "
Nov 03 23:41:42.582393 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.582369    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca" pod "110009d6-c3ce-4f74-9c0b-fcbddfc390f4" (UID: "110009d6-c3ce-4f74-9c0b-fcbddfc390f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:41:42.582531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.582510    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:42.583962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.583937    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca" pod "110009d6-c3ce-4f74-9c0b-fcbddfc390f4" (UID: "110009d6-c3ce-4f74-9c0b-fcbddfc390f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:41:42.584227 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.584212    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl" pod "110009d6-c3ce-4f74-9c0b-fcbddfc390f4" (UID: "110009d6-c3ce-4f74-9c0b-fcbddfc390f4"). InnerVolumeSpecName "kube-api-access-7m2sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:41:42.684052 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.684020    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:42.684052 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.684053    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2sl\" (UniqueName: \"kubernetes.io/projected/110009d6-c3ce-4f74-9c0b-fcbddfc390f4-kube-api-access-7m2sl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:42.818853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.806147    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 23:41:42.818853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:42.806223    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 23:41:43.273179 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.273140    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"
Nov 03 23:41:43.273642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.273180    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v" event={"ID":"110009d6-c3ce-4f74-9c0b-fcbddfc390f4","Type":"ContainerDied","Data":"5a84aee70511a4f08fe117e612fea4bfa80c49a6cfcd6910984160b6f1f87f06"}
Nov 03 23:41:43.273642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.273235    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a84aee70511a4f08fe117e612fea4bfa80c49a6cfcd6910984160b6f1f87f06"
Nov 03 23:41:43.275300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.275265    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc" event={"ID":"177ba727-56bb-496a-82a7-4bafe41aab8a","Type":"ContainerStarted","Data":"b12e6da84501f9b8558afb85088cad7b5eb217493533138196471a8e2180e12f"}
Nov 03 23:41:43.275439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.275311    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc" event={"ID":"177ba727-56bb-496a-82a7-4bafe41aab8a","Type":"ContainerStarted","Data":"510087b9a17ccc3296463405f21e477779c10ea27a949132a730a17d07168e1d"}
Nov 03 23:41:43.297477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.297420    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc" podStartSLOduration=2.297408273 podStartE2EDuration="2.297408273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:43.295238323 +0000 UTC m=+768.394838478" watchObservedRunningTime="2026-11-03 23:41:43.297408273 +0000 UTC m=+768.397008417"
Nov 03 23:41:43.828760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.828297    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 23:41:43.828760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:43.828366    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 23:41:44.284254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:44.284219    5074 generic.go:356] "Generic (PLEG): container finished" podID="177ba727-56bb-496a-82a7-4bafe41aab8a" containerID="b12e6da84501f9b8558afb85088cad7b5eb217493533138196471a8e2180e12f" exitCode=0
Nov 03 23:41:44.284652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:44.284306    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc" event={"ID":"177ba727-56bb-496a-82a7-4bafe41aab8a","Type":"ContainerDied","Data":"b12e6da84501f9b8558afb85088cad7b5eb217493533138196471a8e2180e12f"}
Nov 03 23:41:45.487232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.487205    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:45.537776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.523674    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm\" (UniqueName: \"kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm\") pod \"177ba727-56bb-496a-82a7-4bafe41aab8a\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") "
Nov 03 23:41:45.537776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.523729    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca\") pod \"177ba727-56bb-496a-82a7-4bafe41aab8a\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") "
Nov 03 23:41:45.537776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.523945    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca\") pod \"177ba727-56bb-496a-82a7-4bafe41aab8a\" (UID: \"177ba727-56bb-496a-82a7-4bafe41aab8a\") "
Nov 03 23:41:45.537776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.536278    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca" pod "177ba727-56bb-496a-82a7-4bafe41aab8a" (UID: "177ba727-56bb-496a-82a7-4bafe41aab8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:41:45.537776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.537486    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca" pod "177ba727-56bb-496a-82a7-4bafe41aab8a" (UID: "177ba727-56bb-496a-82a7-4bafe41aab8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:41:45.547931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.547864    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm" pod "177ba727-56bb-496a-82a7-4bafe41aab8a" (UID: "177ba727-56bb-496a-82a7-4bafe41aab8a"). InnerVolumeSpecName "kube-api-access-75dfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:41:45.602084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.602042    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:45.602468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.602438    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:45.603467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.603445    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:41:45.625496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.625466    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:45.625496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.625493    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177ba727-56bb-496a-82a7-4bafe41aab8a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:45.625496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:45.625504    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dfm\" (UniqueName: \"kubernetes.io/projected/177ba727-56bb-496a-82a7-4bafe41aab8a-kube-api-access-75dfm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:46.298338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:46.298309    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"
Nov 03 23:41:46.298338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:46.298326    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc" event={"ID":"177ba727-56bb-496a-82a7-4bafe41aab8a","Type":"ContainerDied","Data":"510087b9a17ccc3296463405f21e477779c10ea27a949132a730a17d07168e1d"}
Nov 03 23:41:46.298588 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:46.298383    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="510087b9a17ccc3296463405f21e477779c10ea27a949132a730a17d07168e1d"
Nov 03 23:41:50.671049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671014    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"]
Nov 03 23:41:50.671495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671482    5074 cpu_manager.go:414] "containerMap: removing container" podUID="110009d6-c3ce-4f74-9c0b-fcbddfc390f4" containerName="revoke-keys"
Nov 03 23:41:50.671549 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671496    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110009d6-c3ce-4f74-9c0b-fcbddfc390f4" containerName="revoke-keys"
Nov 03 23:41:50.671549 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671525    5074 cpu_manager.go:414] "containerMap: removing container" podUID="177ba727-56bb-496a-82a7-4bafe41aab8a" containerName="revoke-keys"
Nov 03 23:41:50.671549 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671530    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177ba727-56bb-496a-82a7-4bafe41aab8a" containerName="revoke-keys"
Nov 03 23:41:50.671640 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671620    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="110009d6-c3ce-4f74-9c0b-fcbddfc390f4" containerName="revoke-keys"
Nov 03 23:41:50.671640 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.671632    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="177ba727-56bb-496a-82a7-4bafe41aab8a" containerName="revoke-keys"
Nov 03 23:41:50.675323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.675305    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.681788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.681765    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:41:50.681998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.681910    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:41:50.684286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.684156    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"]
Nov 03 23:41:50.779399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.779364    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.779573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.779412    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlfs\" (UniqueName: \"kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.779573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.779558    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.880907 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.880875    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.881107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.880915    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlfs\" (UniqueName: \"kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.881107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.880976    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.881647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.881627    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.883354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.883330    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.894285 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.894266    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlfs\" (UniqueName: \"kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs\") pod \"maas-api-revoke-sub-62b574d4-hhh4j\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:50.991280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:50.991253    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:51.276992 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:51.276965    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"]
Nov 03 23:41:51.278532 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:51.278497    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0910b88f_490a_4321_ac53_144426420ead.slice/crio-18ab106e3e7ec88a9fbeeeb86cf2c3e9c56d95e7e53f9212cf59984ab36b49f3 WatchSource:0}: Error finding container 18ab106e3e7ec88a9fbeeeb86cf2c3e9c56d95e7e53f9212cf59984ab36b49f3: Status 404 returned error can't find the container with id 18ab106e3e7ec88a9fbeeeb86cf2c3e9c56d95e7e53f9212cf59984ab36b49f3
Nov 03 23:41:51.341119 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:51.341051    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j" event={"ID":"0910b88f-490a-4321-ac53-144426420ead","Type":"ContainerStarted","Data":"18ab106e3e7ec88a9fbeeeb86cf2c3e9c56d95e7e53f9212cf59984ab36b49f3"}
Nov 03 23:41:52.348882 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.348841    5074 generic.go:356] "Generic (PLEG): container finished" podID="0910b88f-490a-4321-ac53-144426420ead" containerID="651db41aa6bb54c953e53cebce5a68a413bd89ea6052a0ea49456c9d41132f20" exitCode=0
Nov 03 23:41:52.349239 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.348925    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j" event={"ID":"0910b88f-490a-4321-ac53-144426420ead","Type":"ContainerDied","Data":"651db41aa6bb54c953e53cebce5a68a413bd89ea6052a0ea49456c9d41132f20"}
Nov 03 23:41:52.541080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.540997    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"]
Nov 03 23:41:52.545079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.545058    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.554466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.554442    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"]
Nov 03 23:41:52.701844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.701797    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sw9f\" (UniqueName: \"kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.702001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.701962    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.702106 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.702093    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.803220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.803130    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.803220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.803177    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sw9f\" (UniqueName: \"kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.803435 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.803273    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.804061 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.804035    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.805789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.805752    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.817033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.817006    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sw9f\" (UniqueName: \"kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f\") pod \"maas-api-revoke-sub-827fbe14-zvvjv\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:52.859751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:52.859719    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:53.152466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.152436    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"]
Nov 03 23:41:53.154722 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:53.154691    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0036ea50_7f0a_4f60_8bac_e8af2b7235b0.slice/crio-af729298140deda1c992f5c0e26dd26fec99be3cd12fee75bb7fb3efbfaec59a WatchSource:0}: Error finding container af729298140deda1c992f5c0e26dd26fec99be3cd12fee75bb7fb3efbfaec59a: Status 404 returned error can't find the container with id af729298140deda1c992f5c0e26dd26fec99be3cd12fee75bb7fb3efbfaec59a
Nov 03 23:41:53.357160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.357124    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv" event={"ID":"0036ea50-7f0a-4f60-8bac-e8af2b7235b0","Type":"ContainerStarted","Data":"af729298140deda1c992f5c0e26dd26fec99be3cd12fee75bb7fb3efbfaec59a"}
Nov 03 23:41:53.517583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.517351    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:53.613077 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.613045    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs\" (UniqueName: \"kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs\") pod \"0910b88f-490a-4321-ac53-144426420ead\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") "
Nov 03 23:41:53.613265 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.613093    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca\") pod \"0910b88f-490a-4321-ac53-144426420ead\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") "
Nov 03 23:41:53.613550 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.613522    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca" pod "0910b88f-490a-4321-ac53-144426420ead" (UID: "0910b88f-490a-4321-ac53-144426420ead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:41:53.613697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.613671    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca\") pod \"0910b88f-490a-4321-ac53-144426420ead\" (UID: \"0910b88f-490a-4321-ac53-144426420ead\") "
Nov 03 23:41:53.614132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.614108    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0910b88f-490a-4321-ac53-144426420ead-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:53.615873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.615845    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs" pod "0910b88f-490a-4321-ac53-144426420ead" (UID: "0910b88f-490a-4321-ac53-144426420ead"). InnerVolumeSpecName "kube-api-access-7dlfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:41:53.615962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.615903    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca" pod "0910b88f-490a-4321-ac53-144426420ead" (UID: "0910b88f-490a-4321-ac53-144426420ead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:41:53.715078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.715047    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0910b88f-490a-4321-ac53-144426420ead-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:53.715078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:53.715076    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlfs\" (UniqueName: \"kubernetes.io/projected/0910b88f-490a-4321-ac53-144426420ead-kube-api-access-7dlfs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:54.364891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:54.364802    5074 generic.go:356] "Generic (PLEG): container finished" podID="0036ea50-7f0a-4f60-8bac-e8af2b7235b0" containerID="7ef55a1453fc6c3a43ac312044e23e9a065b39ed069e3544bf666e1ad1a917b0" exitCode=0
Nov 03 23:41:54.365349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:54.364885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv" event={"ID":"0036ea50-7f0a-4f60-8bac-e8af2b7235b0","Type":"ContainerDied","Data":"7ef55a1453fc6c3a43ac312044e23e9a065b39ed069e3544bf666e1ad1a917b0"}
Nov 03 23:41:54.366471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:54.366453    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"
Nov 03 23:41:54.366561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:54.366456    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j" event={"ID":"0910b88f-490a-4321-ac53-144426420ead","Type":"ContainerDied","Data":"18ab106e3e7ec88a9fbeeeb86cf2c3e9c56d95e7e53f9212cf59984ab36b49f3"}
Nov 03 23:41:54.366598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:54.366564    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ab106e3e7ec88a9fbeeeb86cf2c3e9c56d95e7e53f9212cf59984ab36b49f3"
Nov 03 23:41:55.547989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.547959    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:55.634515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.634424    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca\") pod \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") "
Nov 03 23:41:55.634676 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.634563    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca\") pod \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") "
Nov 03 23:41:55.634676 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.634622    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f\" (UniqueName: \"kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f\") pod \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\" (UID: \"0036ea50-7f0a-4f60-8bac-e8af2b7235b0\") "
Nov 03 23:41:55.634994 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.634963    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca" pod "0036ea50-7f0a-4f60-8bac-e8af2b7235b0" (UID: "0036ea50-7f0a-4f60-8bac-e8af2b7235b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:41:55.636517 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.636494    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca" pod "0036ea50-7f0a-4f60-8bac-e8af2b7235b0" (UID: "0036ea50-7f0a-4f60-8bac-e8af2b7235b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:41:55.636644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.636623    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f" pod "0036ea50-7f0a-4f60-8bac-e8af2b7235b0" (UID: "0036ea50-7f0a-4f60-8bac-e8af2b7235b0"). InnerVolumeSpecName "kube-api-access-2sw9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:41:55.736139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.736107    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:55.736139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.736137    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sw9f\" (UniqueName: \"kubernetes.io/projected/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-kube-api-access-2sw9f\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:55.736139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:55.736147    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0036ea50-7f0a-4f60-8bac-e8af2b7235b0-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:41:56.379484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:56.379436    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"
Nov 03 23:41:56.379659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:56.379474    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv" event={"ID":"0036ea50-7f0a-4f60-8bac-e8af2b7235b0","Type":"ContainerDied","Data":"af729298140deda1c992f5c0e26dd26fec99be3cd12fee75bb7fb3efbfaec59a"}
Nov 03 23:41:56.379659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:56.379528    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af729298140deda1c992f5c0e26dd26fec99be3cd12fee75bb7fb3efbfaec59a"
Nov 03 23:41:58.049792 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:41:58.049720    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:41:58.285584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.285546    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"]
Nov 03 23:41:58.286128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.286109    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0910b88f-490a-4321-ac53-144426420ead" containerName="revoke-keys"
Nov 03 23:41:58.286128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.286129    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0910b88f-490a-4321-ac53-144426420ead" containerName="revoke-keys"
Nov 03 23:41:58.286279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.286148    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0036ea50-7f0a-4f60-8bac-e8af2b7235b0" containerName="revoke-keys"
Nov 03 23:41:58.286279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.286155    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0036ea50-7f0a-4f60-8bac-e8af2b7235b0" containerName="revoke-keys"
Nov 03 23:41:58.286370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.286301    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0910b88f-490a-4321-ac53-144426420ead" containerName="revoke-keys"
Nov 03 23:41:58.286370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.286316    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0036ea50-7f0a-4f60-8bac-e8af2b7235b0" containerName="revoke-keys"
Nov 03 23:41:58.289777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.289756    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.300198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300138    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5ef40e0b1eca\""
Nov 03 23:41:58.300332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300221    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 23:41:58.300332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300284    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 23:41:58.300443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300355    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 23:41:58.300443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300387    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 23:41:58.300443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300438    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2726b4a083b9\""
Nov 03 23:41:58.300593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.300539    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cvmwv\""
Nov 03 23:41:58.301447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.301426    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"]
Nov 03 23:41:58.373822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.373734    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.374002 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.373935    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5k85\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.374002 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.373993    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.374152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.374036    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.474949 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.474909    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.475133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.474985    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5k85\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.475133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.475032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.475133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.475080    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.476060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.476032    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.476183 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.476116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.477787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.477761    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.489380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.489344    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5k85\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85\") pod \"payload-processing-external-model-785c9966b6-qsrjq\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") " pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.605006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.604915    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:58.903829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:58.903664    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"]
Nov 03 23:41:58.905728 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:41:58.905698    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1319f9b3_8a66_4e3e_b6b2_38195e0e4622.slice/crio-ae02b2a3da716ca108db67f6cecb70264885eb504cb2e77f0d24011e26c11e2f WatchSource:0}: Error finding container ae02b2a3da716ca108db67f6cecb70264885eb504cb2e77f0d24011e26c11e2f: Status 404 returned error can't find the container with id ae02b2a3da716ca108db67f6cecb70264885eb504cb2e77f0d24011e26c11e2f
Nov 03 23:41:59.404051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:59.404007    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq" event={"ID":"1319f9b3-8a66-4e3e-b6b2-38195e0e4622","Type":"ContainerStarted","Data":"59d6cf9be8faded690e198eac961a4bbdf7a3c817705c0d794cb044f675af36b"}
Nov 03 23:41:59.404593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:59.404078    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq" event={"ID":"1319f9b3-8a66-4e3e-b6b2-38195e0e4622","Type":"ContainerStarted","Data":"ae02b2a3da716ca108db67f6cecb70264885eb504cb2e77f0d24011e26c11e2f"}
Nov 03 23:41:59.404593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:59.404325    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:41:59.428396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:41:59.428326    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq" podStartSLOduration=1.428307288 podStartE2EDuration="1.428307288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:59.425156383 +0000 UTC m=+784.524756540" watchObservedRunningTime="2026-11-03 23:41:59.428307288 +0000 UTC m=+784.527907432"
Nov 03 23:42:01.915354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:01.915322    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"]
Nov 03 23:42:01.919509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:01.919491    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:01.926976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:01.926056    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:42:01.926976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:01.926056    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:42:01.928039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:01.928018    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"]
Nov 03 23:42:02.026213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.026176    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppj44\" (UniqueName: \"kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.026388 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.026246    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.026430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.026404    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.127856 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.127821    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.128024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.127958    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppj44\" (UniqueName: \"kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.128112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.128023    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.128678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.128655    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.130365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.130341    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.141509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.141481    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppj44\" (UniqueName: \"kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44\") pod \"maas-api-revoke-sub-d7192de9-jbbj9\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.238447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.238416    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:02.520346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.520312    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"]
Nov 03 23:42:02.923407 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.923377    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"]
Nov 03 23:42:02.927629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.927611    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:02.936373 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:02.935959    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"]
Nov 03 23:42:03.044387 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.044341    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.044696 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.044675    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq47l\" (UniqueName: \"kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.044861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.044840    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.146078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.146035    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.146320 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.146299    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.146566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.146542    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq47l\" (UniqueName: \"kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.146897 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.146870    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.148974 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.148951    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.159951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.159890    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq47l\" (UniqueName: \"kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l\") pod \"maas-api-revoke-sub-da6d203e-7h9t4\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.243755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.243708    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:03.453795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.453750    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9" event={"ID":"d0185379-8e2f-4f3c-b9dc-bae20ac7f718","Type":"ContainerStarted","Data":"64b73f19ec402026228501226a9011aba2314ed3469d2d4881ee8646e675a585"}
Nov 03 23:42:03.454003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.453842    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9" event={"ID":"d0185379-8e2f-4f3c-b9dc-bae20ac7f718","Type":"ContainerStarted","Data":"a1d9e866ec22168ca97f83cb5b6efc355935a3b16518dcba7afbdf7937cf2966"}
Nov 03 23:42:03.475507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.475451    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9" podStartSLOduration=2.475435999 podStartE2EDuration="2.475435999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:03.473385506 +0000 UTC m=+788.572985661" watchObservedRunningTime="2026-11-03 23:42:03.475435999 +0000 UTC m=+788.575036143"
Nov 03 23:42:03.531992 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:03.531957    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"]
Nov 03 23:42:04.462241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:04.462203    5074 generic.go:356] "Generic (PLEG): container finished" podID="d0185379-8e2f-4f3c-b9dc-bae20ac7f718" containerID="64b73f19ec402026228501226a9011aba2314ed3469d2d4881ee8646e675a585" exitCode=0
Nov 03 23:42:04.462934 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:04.462292    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9" event={"ID":"d0185379-8e2f-4f3c-b9dc-bae20ac7f718","Type":"ContainerDied","Data":"64b73f19ec402026228501226a9011aba2314ed3469d2d4881ee8646e675a585"}
Nov 03 23:42:04.465095 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:04.465061    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4" event={"ID":"168723c6-54e6-426b-ac7e-4e596efd2a31","Type":"ContainerStarted","Data":"4992ce638361b4f733b33c5b2c7aa50815e3ad78f90503502fcdde0fcf913ded"}
Nov 03 23:42:04.465201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:04.465108    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4" event={"ID":"168723c6-54e6-426b-ac7e-4e596efd2a31","Type":"ContainerStarted","Data":"f2522ac4e7d593c924961a2d9099038497a037e6a0c193ebcecc0bad3f8e166c"}
Nov 03 23:42:04.511397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:04.511135    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4" podStartSLOduration=2.511118311 podStartE2EDuration="2.511118311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:04.510163398 +0000 UTC m=+789.609763544" watchObservedRunningTime="2026-11-03 23:42:04.511118311 +0000 UTC m=+789.610718455"
Nov 03 23:42:05.498188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:05.498149    5074 generic.go:356] "Generic (PLEG): container finished" podID="168723c6-54e6-426b-ac7e-4e596efd2a31" containerID="4992ce638361b4f733b33c5b2c7aa50815e3ad78f90503502fcdde0fcf913ded" exitCode=0
Nov 03 23:42:05.499289 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:05.499260    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4" event={"ID":"168723c6-54e6-426b-ac7e-4e596efd2a31","Type":"ContainerDied","Data":"4992ce638361b4f733b33c5b2c7aa50815e3ad78f90503502fcdde0fcf913ded"}
Nov 03 23:42:06.389632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.389602    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:06.545761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.545715    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"
Nov 03 23:42:06.547267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.545824    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9" event={"ID":"d0185379-8e2f-4f3c-b9dc-bae20ac7f718","Type":"ContainerDied","Data":"a1d9e866ec22168ca97f83cb5b6efc355935a3b16518dcba7afbdf7937cf2966"}
Nov 03 23:42:06.547267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.547218    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d9e866ec22168ca97f83cb5b6efc355935a3b16518dcba7afbdf7937cf2966"
Nov 03 23:42:06.562062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.561218    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca\") pod \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") "
Nov 03 23:42:06.562062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.561257    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca\") pod \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") "
Nov 03 23:42:06.562062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.561323    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44\" (UniqueName: \"kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44\") pod \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\" (UID: \"d0185379-8e2f-4f3c-b9dc-bae20ac7f718\") "
Nov 03 23:42:06.562062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.561926    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca" pod "d0185379-8e2f-4f3c-b9dc-bae20ac7f718" (UID: "d0185379-8e2f-4f3c-b9dc-bae20ac7f718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:06.564495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.564453    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca" pod "d0185379-8e2f-4f3c-b9dc-bae20ac7f718" (UID: "d0185379-8e2f-4f3c-b9dc-bae20ac7f718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:42:06.570933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.570893    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44" pod "d0185379-8e2f-4f3c-b9dc-bae20ac7f718" (UID: "d0185379-8e2f-4f3c-b9dc-bae20ac7f718"). InnerVolumeSpecName "kube-api-access-ppj44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:06.663354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.663216    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppj44\" (UniqueName: \"kubernetes.io/projected/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-kube-api-access-ppj44\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:06.663354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.663255    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:06.663354 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.663289    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0185379-8e2f-4f3c-b9dc-bae20ac7f718-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:06.703478 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.703449    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:06.764269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.764234    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l\" (UniqueName: \"kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l\") pod \"168723c6-54e6-426b-ac7e-4e596efd2a31\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") "
Nov 03 23:42:06.764464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.764430    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca\") pod \"168723c6-54e6-426b-ac7e-4e596efd2a31\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") "
Nov 03 23:42:06.764529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.764467    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca\") pod \"168723c6-54e6-426b-ac7e-4e596efd2a31\" (UID: \"168723c6-54e6-426b-ac7e-4e596efd2a31\") "
Nov 03 23:42:06.764970 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.764936    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca" pod "168723c6-54e6-426b-ac7e-4e596efd2a31" (UID: "168723c6-54e6-426b-ac7e-4e596efd2a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:06.766680 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.766655    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l" pod "168723c6-54e6-426b-ac7e-4e596efd2a31" (UID: "168723c6-54e6-426b-ac7e-4e596efd2a31"). InnerVolumeSpecName "kube-api-access-lq47l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:06.767090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.767073    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca" pod "168723c6-54e6-426b-ac7e-4e596efd2a31" (UID: "168723c6-54e6-426b-ac7e-4e596efd2a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:42:06.865329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.865299    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq47l\" (UniqueName: \"kubernetes.io/projected/168723c6-54e6-426b-ac7e-4e596efd2a31-kube-api-access-lq47l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:06.865329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.865324    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:06.865329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:06.865334    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168723c6-54e6-426b-ac7e-4e596efd2a31-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:07.552809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:07.552705    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4" event={"ID":"168723c6-54e6-426b-ac7e-4e596efd2a31","Type":"ContainerDied","Data":"f2522ac4e7d593c924961a2d9099038497a037e6a0c193ebcecc0bad3f8e166c"}
Nov 03 23:42:07.552809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:07.552729    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"
Nov 03 23:42:07.552809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:07.552774    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2522ac4e7d593c924961a2d9099038497a037e6a0c193ebcecc0bad3f8e166c"
Nov 03 23:42:10.433546 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:10.433514    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:42:15.186553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.186504    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"]
Nov 03 23:42:15.188638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.188615    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d0185379-8e2f-4f3c-b9dc-bae20ac7f718" containerName="revoke-keys"
Nov 03 23:42:15.188803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.188787    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0185379-8e2f-4f3c-b9dc-bae20ac7f718" containerName="revoke-keys"
Nov 03 23:42:15.189003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.188989    5074 cpu_manager.go:414] "containerMap: removing container" podUID="168723c6-54e6-426b-ac7e-4e596efd2a31" containerName="revoke-keys"
Nov 03 23:42:15.189105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.189092    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168723c6-54e6-426b-ac7e-4e596efd2a31" containerName="revoke-keys"
Nov 03 23:42:15.189451 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.189433    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="168723c6-54e6-426b-ac7e-4e596efd2a31" containerName="revoke-keys"
Nov 03 23:42:15.189580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.189567    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0185379-8e2f-4f3c-b9dc-bae20ac7f718" containerName="revoke-keys"
Nov 03 23:42:15.203946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.203922    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"]
Nov 03 23:42:15.204211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.204150    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.234937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.234895    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-23e553f05ab2\""
Nov 03 23:42:15.235457 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.235427    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-07e468a329ab\""
Nov 03 23:42:15.339124 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.339082    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.339455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.339428    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.339645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.339625    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmtj\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.339797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.339779    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.442396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.442309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmtj\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.442637 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.442614    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.442933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.442912    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.443098 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.443079    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.444058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.444033    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.462339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.444925    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.465384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.465358    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmtj\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.494158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.491759    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials\") pod \"payload-processing-external-model-5d887c9bbc-m7zcc\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") " pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:15.605947 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:15.605909    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:16.097305 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:16.097144    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"]
Nov 03 23:42:16.698217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:16.698174    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc" event={"ID":"c2326428-8e4f-48ed-847b-efdfed20a2a9","Type":"ContainerStarted","Data":"4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56"}
Nov 03 23:42:16.698862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:16.698236    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc" event={"ID":"c2326428-8e4f-48ed-847b-efdfed20a2a9","Type":"ContainerStarted","Data":"f07fc6196cf00910e50c6acc1fd65ed3458eb0f21c647f0b5232594497caad4e"}
Nov 03 23:42:16.698862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:16.698257    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:16.720484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:16.720408    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc" podStartSLOduration=2.720385969 podStartE2EDuration="2.720385969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:16.71878641 +0000 UTC m=+801.818386553" watchObservedRunningTime="2026-11-03 23:42:16.720385969 +0000 UTC m=+801.819986113"
Nov 03 23:42:18.946761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:18.946706    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"]
Nov 03 23:42:18.952808 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:18.952356    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:18.966924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:18.962549    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\""
Nov 03 23:42:18.968871 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:18.968842    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"]
Nov 03 23:42:19.020108 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020076    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020114    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020206    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020259    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020397    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020552 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020464    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfwm\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020552 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020537    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.020645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.020568    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121351    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121444    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfwm\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121480    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121501    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121650    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121704    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121774    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121848    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.121955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.121937    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.122061 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.122013    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.122344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.122301    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.122344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.122319    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.122633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.122509    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.122633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.122578    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.124387 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.124361    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.124574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.124553    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.134414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.134369    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.134928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.134888    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfwm\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.286028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.285940    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:19.584354 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:42:19.584316    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9fd0ee_6312_4c25_a52b_05d4f0e886d5.slice/crio-b37caa320e893f21e09f07190dfffbaa46dce1b6893322e043da0e9e064ac166 WatchSource:0}: Error finding container b37caa320e893f21e09f07190dfffbaa46dce1b6893322e043da0e9e064ac166: Status 404 returned error can't find the container with id b37caa320e893f21e09f07190dfffbaa46dce1b6893322e043da0e9e064ac166
Nov 03 23:42:19.587840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.587810    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"]
Nov 03 23:42:19.612764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.595455    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:42:19.612764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.595631    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:42:19.612764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.595683    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:42:19.733979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.733774    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" event={"ID":"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5","Type":"ContainerStarted","Data":"ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af"}
Nov 03 23:42:19.733979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:19.733845    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" event={"ID":"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5","Type":"ContainerStarted","Data":"b37caa320e893f21e09f07190dfffbaa46dce1b6893322e043da0e9e064ac166"}
Nov 03 23:42:20.794259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:20.794206    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podStartSLOduration=2.7941887100000002 podStartE2EDuration="2.79418871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:20.792385752 +0000 UTC m=+805.891985909" watchObservedRunningTime="2026-11-03 23:42:20.79418871 +0000 UTC m=+805.893788856"
Nov 03 23:42:21.286475 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:21.286431    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:21.288512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:21.288479    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 23:42:21.288667 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:21.288550    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 23:42:22.301776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:22.299161    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 23:42:22.301776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:22.299227    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 23:42:23.289184 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:23.289148    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 23:42:23.289448 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:23.289425    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 23:42:24.290281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:24.290245    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 23:42:24.290910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:24.290880    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 23:42:25.288050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:25.288003    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 23:42:25.288248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:25.288082    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 23:42:26.286812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.286772    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 23:42:26.287255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.286854    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 23:42:26.580083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.579987    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"]
Nov 03 23:42:26.589374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.589342    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.610883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.610717    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:42:26.610883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.610804    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:42:26.613450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.613413    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"]
Nov 03 23:42:26.734999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.734959    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rg68\" (UniqueName: \"kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.735202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.735007    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.735274 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.735227    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.836540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.836438    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.836768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.836606    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rg68\" (UniqueName: \"kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.836768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.836658    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.837381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.837356    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.839676 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.839651    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.851419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.851382    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rg68\" (UniqueName: \"kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68\") pod \"maas-api-revoke-sub-c0a34131-cz9js\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:26.904161 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:26.904123    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:27.204660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.204626    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"]
Nov 03 23:42:27.206824 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:42:27.206792    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146fdda6_d0c2_4267_8c5f_cd313132060d.slice/crio-678f2b057355e382b9c29313af963f199b674978771deaab61e1683ee0704e56 WatchSource:0}: Error finding container 678f2b057355e382b9c29313af963f199b674978771deaab61e1683ee0704e56: Status 404 returned error can't find the container with id 678f2b057355e382b9c29313af963f199b674978771deaab61e1683ee0704e56
Nov 03 23:42:27.291111 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.291079    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:27.291548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.291495    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:27.292480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.292425    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:42:27.710536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.710508    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:27.768414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.768308    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"]
Nov 03 23:42:27.988815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.988781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js" event={"ID":"146fdda6-d0c2-4267-8c5f-cd313132060d","Type":"ContainerStarted","Data":"fb1214939d643771630d3dd9beabd9d41cf2cd6ffad64d3fc7fa8b16551ae99f"}
Nov 03 23:42:27.988992 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:27.988842    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js" event={"ID":"146fdda6-d0c2-4267-8c5f-cd313132060d","Type":"ContainerStarted","Data":"678f2b057355e382b9c29313af963f199b674978771deaab61e1683ee0704e56"}
Nov 03 23:42:28.019383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:28.016217    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js" podStartSLOduration=2.016192586 podStartE2EDuration="2.016192586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:28.009303053 +0000 UTC m=+813.108903199" watchObservedRunningTime="2026-11-03 23:42:28.016192586 +0000 UTC m=+813.115792753"
Nov 03 23:42:28.036514 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:28.036459    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:42:28.219926 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:28.219889    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146fdda6_d0c2_4267_8c5f_cd313132060d.slice/crio-conmon-fb1214939d643771630d3dd9beabd9d41cf2cd6ffad64d3fc7fa8b16551ae99f.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 23:42:28.998729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:28.998686    5074 generic.go:356] "Generic (PLEG): container finished" podID="146fdda6-d0c2-4267-8c5f-cd313132060d" containerID="fb1214939d643771630d3dd9beabd9d41cf2cd6ffad64d3fc7fa8b16551ae99f" exitCode=0
Nov 03 23:42:28.999208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:28.998771    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js" event={"ID":"146fdda6-d0c2-4267-8c5f-cd313132060d","Type":"ContainerDied","Data":"fb1214939d643771630d3dd9beabd9d41cf2cd6ffad64d3fc7fa8b16551ae99f"}
Nov 03 23:42:30.160345 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.160319    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:30.283987 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.283904    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca\") pod \"146fdda6-d0c2-4267-8c5f-cd313132060d\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") "
Nov 03 23:42:30.284141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.284110    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68\" (UniqueName: \"kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68\") pod \"146fdda6-d0c2-4267-8c5f-cd313132060d\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") "
Nov 03 23:42:30.284141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.284135    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca\") pod \"146fdda6-d0c2-4267-8c5f-cd313132060d\" (UID: \"146fdda6-d0c2-4267-8c5f-cd313132060d\") "
Nov 03 23:42:30.284541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.284514    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca" pod "146fdda6-d0c2-4267-8c5f-cd313132060d" (UID: "146fdda6-d0c2-4267-8c5f-cd313132060d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:30.286006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.285972    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca" pod "146fdda6-d0c2-4267-8c5f-cd313132060d" (UID: "146fdda6-d0c2-4267-8c5f-cd313132060d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:42:30.286120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.286104    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68" pod "146fdda6-d0c2-4267-8c5f-cd313132060d" (UID: "146fdda6-d0c2-4267-8c5f-cd313132060d"). InnerVolumeSpecName "kube-api-access-7rg68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:30.385054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.385018    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rg68\" (UniqueName: \"kubernetes.io/projected/146fdda6-d0c2-4267-8c5f-cd313132060d-kube-api-access-7rg68\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:30.385054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.385046    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:30.385054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:30.385060    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146fdda6-d0c2-4267-8c5f-cd313132060d-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:31.017855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:31.017825    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"
Nov 03 23:42:31.024698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:31.023537    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js" event={"ID":"146fdda6-d0c2-4267-8c5f-cd313132060d","Type":"ContainerDied","Data":"678f2b057355e382b9c29313af963f199b674978771deaab61e1683ee0704e56"}
Nov 03 23:42:31.024698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:31.023604    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="678f2b057355e382b9c29313af963f199b674978771deaab61e1683ee0704e56"
Nov 03 23:42:32.226766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.221277    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"]
Nov 03 23:42:32.262149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.262116    5074 cpu_manager.go:414] "containerMap: removing container" podUID="146fdda6-d0c2-4267-8c5f-cd313132060d" containerName="revoke-keys"
Nov 03 23:42:32.368824 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.368784    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146fdda6-d0c2-4267-8c5f-cd313132060d" containerName="revoke-keys"
Nov 03 23:42:32.488851 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.487341    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="146fdda6-d0c2-4267-8c5f-cd313132060d" containerName="revoke-keys"
Nov 03 23:42:32.612120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.612084    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"]
Nov 03 23:42:32.612609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.612590    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.672760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.672254    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5cb349-openshift-default-dockercfg-qzhc9\""
Nov 03 23:42:32.673323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.673213    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.673323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.673289    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.673510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.673365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.673510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.673401    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.673510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.673449    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.673510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.673504    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.674605 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.674576    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.674892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.674871    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.676616 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.676591    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87shf\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.770923 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.770843    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq" podUID="1319f9b3-8a66-4e3e-b6b2-38195e0e4622" containerName="payload-processing" containerID="cri-o://59d6cf9be8faded690e198eac961a4bbdf7a3c817705c0d794cb044f675af36b" gracePeriod=25
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783046    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783114    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783169    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783213    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87shf\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783272    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783315    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783376    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783404    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.784758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.783449    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.786321 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.786282    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.789150 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.789117    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.789559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.789542    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.795922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.790214    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.818336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.818308    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.823375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.823352    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87shf\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.831396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.825266    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.842775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.842049    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:32.867775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:32.850754    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:33.085951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.085915    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:33.167765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.158152    5074 generic.go:356] "Generic (PLEG): container finished" podID="1319f9b3-8a66-4e3e-b6b2-38195e0e4622" containerID="59d6cf9be8faded690e198eac961a4bbdf7a3c817705c0d794cb044f675af36b" exitCode=0
Nov 03 23:42:33.167765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.158328    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq" event={"ID":"1319f9b3-8a66-4e3e-b6b2-38195e0e4622","Type":"ContainerDied","Data":"59d6cf9be8faded690e198eac961a4bbdf7a3c817705c0d794cb044f675af36b"}
Nov 03 23:42:33.779822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.779794    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:42:33.914812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.908192    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"]
Nov 03 23:42:33.925117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.922424    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume" pod "1319f9b3-8a66-4e3e-b6b2-38195e0e4622" (UID: "1319f9b3-8a66-4e3e-b6b2-38195e0e4622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:33.925117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.923503    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume\") pod \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") "
Nov 03 23:42:33.937760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.932852    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials\") pod \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") "
Nov 03 23:42:33.937760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.932938    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume\") pod \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") "
Nov 03 23:42:33.937760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.932995    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85\") pod \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\" (UID: \"1319f9b3-8a66-4e3e-b6b2-38195e0e4622\") "
Nov 03 23:42:33.937760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.933526    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:33.971846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.960983    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume" pod "1319f9b3-8a66-4e3e-b6b2-38195e0e4622" (UID: "1319f9b3-8a66-4e3e-b6b2-38195e0e4622"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:33.988293 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:42:33.987871    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d1dc37_2703_49d4_852a_b0d3cc3cdc3f.slice/crio-a0ea4482b00759847cddc6aad480484762c83d2d7bd4fe8c6cb4834ba1dfb666 WatchSource:0}: Error finding container a0ea4482b00759847cddc6aad480484762c83d2d7bd4fe8c6cb4834ba1dfb666: Status 404 returned error can't find the container with id a0ea4482b00759847cddc6aad480484762c83d2d7bd4fe8c6cb4834ba1dfb666
Nov 03 23:42:33.988293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.987899    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85" pod "1319f9b3-8a66-4e3e-b6b2-38195e0e4622" (UID: "1319f9b3-8a66-4e3e-b6b2-38195e0e4622"). InnerVolumeSpecName "kube-api-access-h5k85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:33.991548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.990623    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:42:33.991548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.990818    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:42:33.991548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:33.990879    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:42:34.055783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.045627    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:34.055783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.045662    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5k85\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-kube-api-access-h5k85\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:34.117080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.110519    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials" pod "1319f9b3-8a66-4e3e-b6b2-38195e0e4622" (UID: "1319f9b3-8a66-4e3e-b6b2-38195e0e4622"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:34.156357 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.154726    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1319f9b3-8a66-4e3e-b6b2-38195e0e4622-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:34.208778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.196357    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" event={"ID":"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f","Type":"ContainerStarted","Data":"a0ea4482b00759847cddc6aad480484762c83d2d7bd4fe8c6cb4834ba1dfb666"}
Nov 03 23:42:34.208778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.198718    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq" event={"ID":"1319f9b3-8a66-4e3e-b6b2-38195e0e4622","Type":"ContainerDied","Data":"ae02b2a3da716ca108db67f6cecb70264885eb504cb2e77f0d24011e26c11e2f"}
Nov 03 23:42:34.208778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.198799    5074 scope.go:122] "RemoveContainer" containerID="59d6cf9be8faded690e198eac961a4bbdf7a3c817705c0d794cb044f675af36b"
Nov 03 23:42:34.208778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.202494    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"
Nov 03 23:42:34.370462 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.370426    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"]
Nov 03 23:42:34.370711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:34.370693    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-785c9966b6-qsrjq"]
Nov 03 23:42:35.058037 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:35.058000    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1319f9b3-8a66-4e3e-b6b2-38195e0e4622" path="/var/lib/kubelet/pods/1319f9b3-8a66-4e3e-b6b2-38195e0e4622/volumes"
Nov 03 23:42:35.238458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:35.238421    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" event={"ID":"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f","Type":"ContainerStarted","Data":"842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116"}
Nov 03 23:42:35.338823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:35.337040    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podStartSLOduration=3.3370192960000002 podStartE2EDuration="3.337019296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:35.271129136 +0000 UTC m=+820.370729292" watchObservedRunningTime="2026-11-03 23:42:35.337019296 +0000 UTC m=+820.436619441"
Nov 03 23:42:36.101763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.099722    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:36.105756 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.104811    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 23:42:36.105756 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.104876    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 23:42:36.436426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.436389    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"]
Nov 03 23:42:36.437004 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.436984    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1319f9b3-8a66-4e3e-b6b2-38195e0e4622" containerName="payload-processing"
Nov 03 23:42:36.437102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.437006    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1319f9b3-8a66-4e3e-b6b2-38195e0e4622" containerName="payload-processing"
Nov 03 23:42:36.437201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.437187    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1319f9b3-8a66-4e3e-b6b2-38195e0e4622" containerName="payload-processing"
Nov 03 23:42:36.441863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.441843    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.442386 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.442235    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"]
Nov 03 23:42:36.576160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.576121    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.576309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.576204    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xvk\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.576309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.576301    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.576395 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.576322    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.677674 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.677626    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.677871 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.677759    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xvk\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.677944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.677884    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.677944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.677926    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.678773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.678723    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.678935 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.678913    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.681079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.681058    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.691832 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.691776    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xvk\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk\") pod \"payload-processing-external-model-59db5bb669-cv7r4\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") " pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:36.759014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:36.758981    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:37.082074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.082041    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"]
Nov 03 23:42:37.111190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.110092    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 23:42:37.111190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.110149    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 23:42:37.277129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.274343    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4" event={"ID":"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee","Type":"ContainerStarted","Data":"7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d"}
Nov 03 23:42:37.277129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.274394    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4" event={"ID":"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee","Type":"ContainerStarted","Data":"557fb925dd41379ecdc2aca4057922c6d449f6d46f5a6b29daf368409db42fdc"}
Nov 03 23:42:37.277129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.274435    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:37.298478 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.298408    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4" podStartSLOduration=1.2983902299999999 podStartE2EDuration="1.29839023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:37.295119336 +0000 UTC m=+822.394719495" watchObservedRunningTime="2026-11-03 23:42:37.29839023 +0000 UTC m=+822.397990375"
Nov 03 23:42:37.389610 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.389514    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-23e553f05ab2: secret "agc-cred-23e553f05ab2" not found
Nov 03 23:42:37.389610 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.389557    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4: secret "agc-cred-23e553f05ab2" not found
Nov 03 23:42:37.389902 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.389619    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials podName:2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee nodeName:}" failed. No retries permitted until 2026-11-03 23:42:37.889598553 +0000 UTC m=+822.989198703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials") pod "payload-processing-external-model-59db5bb669-cv7r4" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee") : secret "agc-cred-23e553f05ab2" not found
Nov 03 23:42:37.454550 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.454514    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"]
Nov 03 23:42:37.487086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.487053    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"]
Nov 03 23:42:37.491054 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.491030    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.512700 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.512117    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"]
Nov 03 23:42:37.592300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.592258    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.592490 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.592323    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.592490 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.592381    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8x7\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.592490 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.592433    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.693649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.693617    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.693809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.693668    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.693809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.693703    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8x7\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.693809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.693753    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.693994 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.693904    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2726b4a083b9: secret "agc-cred-2726b4a083b9" not found
Nov 03 23:42:37.693994 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.693930    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5ef40e0b1eca: secret "agc-cred-5ef40e0b1eca" not found
Nov 03 23:42:37.693994 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.693941    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7: [secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:37.694138 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.693998    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials podName:9ad4cb26-d80b-4016-a990-7f5ee321566c nodeName:}" failed. No retries permitted until 2026-11-03 23:42:38.193979418 +0000 UTC m=+823.293579549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials") pod "payload-processing-external-model-597f7dcc87-jnsw7" (UID: "9ad4cb26-d80b-4016-a990-7f5ee321566c") : [secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:37.694477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.694455    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.694620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.694597    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.734209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.734180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8x7\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:37.855762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.848730    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"]
Nov 03 23:42:37.855762 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.849100    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7" podUID="9ad4cb26-d80b-4016-a990-7f5ee321566c"
Nov 03 23:42:37.887275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.887240    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"]
Nov 03 23:42:37.891390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.891373    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:37.901765 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.898161    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-23e553f05ab2: secret "agc-cred-23e553f05ab2" not found
Nov 03 23:42:37.901765 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.898188    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2726b4a083b9: secret "agc-cred-2726b4a083b9" not found
Nov 03 23:42:37.901765 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.898198    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5ef40e0b1eca: secret "agc-cred-5ef40e0b1eca" not found
Nov 03 23:42:37.901765 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.898210    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4: [secret "agc-cred-23e553f05ab2" not found, secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:37.901765 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:37.898266    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials podName:2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee nodeName:}" failed. No retries permitted until 2026-11-03 23:42:38.89824995 +0000 UTC m=+823.997850095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials") pod "payload-processing-external-model-59db5bb669-cv7r4" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee") : [secret "agc-cred-23e553f05ab2" not found, secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:37.936383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:37.936349    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"]
Nov 03 23:42:38.025820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.019488    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.025820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.019621    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvmc\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.025820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.019650    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.025820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.019720    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.102104 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.102068    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 23:42:38.102341 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.102313    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 23:42:38.123439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.123400    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvmc\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.124037 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.124016    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.124203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.124186    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.124441 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.124416    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.125281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.125259    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.126910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.126878    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.138151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.138121    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvmc\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.140507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.140479    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials\") pod \"payload-processing-external-model-7b4ff889d7-p4ndx\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") " pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.227761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.226710    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:38.227761 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.226952    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2726b4a083b9: secret "agc-cred-2726b4a083b9" not found
Nov 03 23:42:38.227761 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.226972    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5ef40e0b1eca: secret "agc-cred-5ef40e0b1eca" not found
Nov 03 23:42:38.227761 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.226983    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7: [secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:38.227761 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.227046    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials podName:9ad4cb26-d80b-4016-a990-7f5ee321566c nodeName:}" failed. No retries permitted until 2026-11-03 23:42:39.227028024 +0000 UTC m=+824.326628162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials") pod "payload-processing-external-model-597f7dcc87-jnsw7" (UID: "9ad4cb26-d80b-4016-a990-7f5ee321566c") : [secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:38.234362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.233864    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:38.280266 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.280194    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:38.291160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.291134    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:38.432225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.432189    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7\") pod \"9ad4cb26-d80b-4016-a990-7f5ee321566c\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") "
Nov 03 23:42:38.432402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.432344    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume\") pod \"9ad4cb26-d80b-4016-a990-7f5ee321566c\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") "
Nov 03 23:42:38.432465 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.432403    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume\") pod \"9ad4cb26-d80b-4016-a990-7f5ee321566c\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") "
Nov 03 23:42:38.434048 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.433989    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume" pod "9ad4cb26-d80b-4016-a990-7f5ee321566c" (UID: "9ad4cb26-d80b-4016-a990-7f5ee321566c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:38.435547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.435521    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume" pod "9ad4cb26-d80b-4016-a990-7f5ee321566c" (UID: "9ad4cb26-d80b-4016-a990-7f5ee321566c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:38.435733 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.435716    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7" pod "9ad4cb26-d80b-4016-a990-7f5ee321566c" (UID: "9ad4cb26-d80b-4016-a990-7f5ee321566c"). InnerVolumeSpecName "kube-api-access-th8x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:38.535293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.534451    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:38.535293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.534484    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ad4cb26-d80b-4016-a990-7f5ee321566c-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:38.535293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.534499    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th8x7\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-kube-api-access-th8x7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:38.557611 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:38.557580    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"]
Nov 03 23:42:38.558626 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:42:38.558599    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486182f8_6936_4ea5_81e6_b21f6b8b060c.slice/crio-fe1904516d828da3eb0a1040a069b3ffc4b66837c7af6a4b8bfb4f47ee9eb5fa WatchSource:0}: Error finding container fe1904516d828da3eb0a1040a069b3ffc4b66837c7af6a4b8bfb4f47ee9eb5fa: Status 404 returned error can't find the container with id fe1904516d828da3eb0a1040a069b3ffc4b66837c7af6a4b8bfb4f47ee9eb5fa
Nov 03 23:42:38.939152 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.939118    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-23e553f05ab2: secret "agc-cred-23e553f05ab2" not found
Nov 03 23:42:38.939152 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.939152    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2726b4a083b9: secret "agc-cred-2726b4a083b9" not found
Nov 03 23:42:38.939360 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.939161    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5ef40e0b1eca: secret "agc-cred-5ef40e0b1eca" not found
Nov 03 23:42:38.939360 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.939174    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4: [secret "agc-cred-23e553f05ab2" not found, secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:38.939360 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:38.939251    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials podName:2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee nodeName:}" failed. No retries permitted until 2026-11-03 23:42:40.939228725 +0000 UTC m=+826.038828851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials") pod "payload-processing-external-model-59db5bb669-cv7r4" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee") : [secret "agc-cred-23e553f05ab2" not found, secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:39.103761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.100184    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 23:42:39.103761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.100244    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 23:42:39.242678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.242500    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials\") pod \"payload-processing-external-model-597f7dcc87-jnsw7\" (UID: \"9ad4cb26-d80b-4016-a990-7f5ee321566c\") " pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:39.242678 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:39.242656    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2726b4a083b9: secret "agc-cred-2726b4a083b9" not found
Nov 03 23:42:39.242678 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:39.242678    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5ef40e0b1eca: secret "agc-cred-5ef40e0b1eca" not found
Nov 03 23:42:39.243249 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:39.242690    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7: [secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:39.243249 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:39.242779    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials podName:9ad4cb26-d80b-4016-a990-7f5ee321566c nodeName:}" failed. No retries permitted until 2026-11-03 23:42:41.242733665 +0000 UTC m=+826.342333798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials") pod "payload-processing-external-model-597f7dcc87-jnsw7" (UID: "9ad4cb26-d80b-4016-a990-7f5ee321566c") : [secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:39.288178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.288147    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx" event={"ID":"486182f8-6936-4ea5-81e6-b21f6b8b060c","Type":"ContainerStarted","Data":"a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b"}
Nov 03 23:42:39.288339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.288189    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"
Nov 03 23:42:39.288339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.288207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx" event={"ID":"486182f8-6936-4ea5-81e6-b21f6b8b060c","Type":"ContainerStarted","Data":"fe1904516d828da3eb0a1040a069b3ffc4b66837c7af6a4b8bfb4f47ee9eb5fa"}
Nov 03 23:42:39.288498 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.288480    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:39.315761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.314507    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx" podStartSLOduration=2.314490298 podStartE2EDuration="2.314490298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:39.313107709 +0000 UTC m=+824.412707854" watchObservedRunningTime="2026-11-03 23:42:39.314490298 +0000 UTC m=+824.414090442"
Nov 03 23:42:39.357649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.357621    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"]
Nov 03 23:42:39.367219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.366696    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-597f7dcc87-jnsw7"]
Nov 03 23:42:39.445132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:39.444968    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ad4cb26-d80b-4016-a990-7f5ee321566c-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:40.103760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:40.103091    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 23:42:40.103760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:40.103153    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 23:42:40.961670 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:40.961634    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-23e553f05ab2: secret "agc-cred-23e553f05ab2" not found
Nov 03 23:42:40.961670 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:40.961667    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2726b4a083b9: secret "agc-cred-2726b4a083b9" not found
Nov 03 23:42:40.961670 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:40.961676    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5ef40e0b1eca: secret "agc-cred-5ef40e0b1eca" not found
Nov 03 23:42:40.962185 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:40.961689    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4: [secret "agc-cred-23e553f05ab2" not found, secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:40.962185 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:40.961770    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials podName:2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee nodeName:}" failed. No retries permitted until 2026-11-03 23:42:44.961751512 +0000 UTC m=+830.061351651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials") pod "payload-processing-external-model-59db5bb669-cv7r4" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee") : [secret "agc-cred-23e553f05ab2" not found, secret "agc-cred-2726b4a083b9" not found, secret "agc-cred-5ef40e0b1eca" not found]
Nov 03 23:42:41.021841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:41.021805    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad4cb26-d80b-4016-a990-7f5ee321566c" path="/var/lib/kubelet/pods/9ad4cb26-d80b-4016-a990-7f5ee321566c/volumes"
Nov 03 23:42:41.098160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:41.098125    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 23:42:41.098310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:41.098187    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 23:42:42.103411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:42.103377    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:42.103862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:42.103773    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:42.104643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:42.104619    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 03 23:42:44.289401 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.289357    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4" podUID="2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" containerName="payload-processing" containerID="cri-o://7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d" gracePeriod=25
Nov 03 23:42:44.578430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.578407    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:44.635207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.635175    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume\") pod \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") "
Nov 03 23:42:44.635497 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.635470    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk\") pod \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") "
Nov 03 23:42:44.635617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.635545    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume\") pod \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") "
Nov 03 23:42:44.635731 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.635714    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials\") pod \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\" (UID: \"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee\") "
Nov 03 23:42:44.637120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.637095    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume" pod "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:44.637940 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.637538    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume" pod "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:44.640999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.640947    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk" pod "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee"). InnerVolumeSpecName "kube-api-access-f2xvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:44.658905 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.645073    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials" pod "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" (UID: "2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:44.737421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.737383    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xvk\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-kube-api-access-f2xvk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:44.737421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.737412    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:44.737421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.737423    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:44.737620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:44.737432    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:45.341721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.341151    5074 generic.go:356] "Generic (PLEG): container finished" podID="2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" containerID="7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d" exitCode=0
Nov 03 23:42:45.341721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.341340    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4" event={"ID":"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee","Type":"ContainerDied","Data":"7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d"}
Nov 03 23:42:45.341721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.341395    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4" event={"ID":"2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee","Type":"ContainerDied","Data":"557fb925dd41379ecdc2aca4057922c6d449f6d46f5a6b29daf368409db42fdc"}
Nov 03 23:42:45.341721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.341431    5074 scope.go:122] "RemoveContainer" containerID="7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d"
Nov 03 23:42:45.341721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.341596    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"
Nov 03 23:42:45.363144 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.363124    5074 scope.go:122] "RemoveContainer" containerID="7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d"
Nov 03 23:42:45.363672 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:45.363607    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d\": container with ID starting with 7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d not found: ID does not exist" containerID="7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d"
Nov 03 23:42:45.363819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.363685    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d"} err="failed to get container status \"7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d\": rpc error: code = NotFound desc = could not find container \"7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d\": container with ID starting with 7dfc4cb04058e01e717ec11515a94009cf3b647579c295c6a6ab051f2311345d not found: ID does not exist"
Nov 03 23:42:45.376250 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.376218    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"]
Nov 03 23:42:45.380922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:45.380894    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59db5bb669-cv7r4"]
Nov 03 23:42:47.049426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:47.048983    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" path="/var/lib/kubelet/pods/2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee/volumes"
Nov 03 23:42:50.299482 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:50.299451    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:42:50.356661 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:50.356627    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"]
Nov 03 23:42:53.088364 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.088322    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"]
Nov 03 23:42:53.089036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.089015    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" containerName="payload-processing"
Nov 03 23:42:53.089036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.089034    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" containerName="payload-processing"
Nov 03 23:42:53.092889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.092850    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2219ab7f-2fd0-4ecb-b14f-42f7d667c9ee" containerName="payload-processing"
Nov 03 23:42:53.125763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.125719    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.131952 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.131823    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"]
Nov 03 23:42:53.134374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.132876    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"openshift-service-ca.crt\""
Nov 03 23:42:53.135197 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.135165    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"default-dockercfg-6wx2m\""
Nov 03 23:42:53.135664 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.135629    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"kube-root-ca.crt\""
Nov 03 23:42:53.135956 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.135939    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"e2e-conflict-b-86358670-kserve-self-signed-certs\""
Nov 03 23:42:53.190299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.190259    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.190299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.190313    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.190559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.190359    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.190559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.190403    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8n4\" (UniqueName: \"kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.190559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.190439    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.190559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.190490    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292285    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292336    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292386    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292426    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8n4\" (UniqueName: \"kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292462    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292516    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292836    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.292976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.292950    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.293454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.293432    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.295918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.295893    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.296545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.296521    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.314823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.314796    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8n4\" (UniqueName: \"kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4\") pod \"e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.481454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.481417    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:53.782681 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:53.782647    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"]
Nov 03 23:42:53.783049 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:42:53.783024    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e26b7c8_1dee_44f5_971d_9531dfdda203.slice/crio-7d68ed734463c8791eeb2ada32c4deba37ff26bbd00975766b7b85671d7f3add WatchSource:0}: Error finding container 7d68ed734463c8791eeb2ada32c4deba37ff26bbd00975766b7b85671d7f3add: Status 404 returned error can't find the container with id 7d68ed734463c8791eeb2ada32c4deba37ff26bbd00975766b7b85671d7f3add
Nov 03 23:42:54.437530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:54.437489    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" event={"ID":"9e26b7c8-1dee-44f5-971d-9531dfdda203","Type":"ContainerStarted","Data":"261a48454147fc538ad45e0b756dc54a533c4390914a2d92475d89ca62491daf"}
Nov 03 23:42:54.437944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:54.437538    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" event={"ID":"9e26b7c8-1dee-44f5-971d-9531dfdda203","Type":"ContainerStarted","Data":"7d68ed734463c8791eeb2ada32c4deba37ff26bbd00975766b7b85671d7f3add"}
Nov 03 23:42:54.457829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:54.457759    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" podStartSLOduration=1.457724111 podStartE2EDuration="1.457724111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:54.457540583 +0000 UTC m=+839.557140724" watchObservedRunningTime="2026-11-03 23:42:54.457724111 +0000 UTC m=+839.557324246"
Nov 03 23:42:54.481920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:54.481887    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:42:55.357343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.357309    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc" podUID="c2326428-8e4f-48ed-847b-efdfed20a2a9" containerName="payload-processing" containerID="cri-o://4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56" gracePeriod=25
Nov 03 23:42:55.643339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.643317    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:55.723095 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.723062    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials\") pod \"c2326428-8e4f-48ed-847b-efdfed20a2a9\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") "
Nov 03 23:42:55.723291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.723187    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume\") pod \"c2326428-8e4f-48ed-847b-efdfed20a2a9\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") "
Nov 03 23:42:55.723291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.723241    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj\") pod \"c2326428-8e4f-48ed-847b-efdfed20a2a9\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") "
Nov 03 23:42:55.723291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.723262    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume\") pod \"c2326428-8e4f-48ed-847b-efdfed20a2a9\" (UID: \"c2326428-8e4f-48ed-847b-efdfed20a2a9\") "
Nov 03 23:42:55.723658 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.723630    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume" pod "c2326428-8e4f-48ed-847b-efdfed20a2a9" (UID: "c2326428-8e4f-48ed-847b-efdfed20a2a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:55.723730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.723653    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume" pod "c2326428-8e4f-48ed-847b-efdfed20a2a9" (UID: "c2326428-8e4f-48ed-847b-efdfed20a2a9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:42:55.725561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.725537    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials" pod "c2326428-8e4f-48ed-847b-efdfed20a2a9" (UID: "c2326428-8e4f-48ed-847b-efdfed20a2a9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:55.725694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.725558    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj" pod "c2326428-8e4f-48ed-847b-efdfed20a2a9" (UID: "c2326428-8e4f-48ed-847b-efdfed20a2a9"). InnerVolumeSpecName "kube-api-access-dbmtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:42:55.824885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.824841    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:55.824885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.824882    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:55.824885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.824893    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmtj\" (UniqueName: \"kubernetes.io/projected/c2326428-8e4f-48ed-847b-efdfed20a2a9-kube-api-access-dbmtj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:55.825125 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:55.824904    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2326428-8e4f-48ed-847b-efdfed20a2a9-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:42:56.452896 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.452860    5074 generic.go:356] "Generic (PLEG): container finished" podID="c2326428-8e4f-48ed-847b-efdfed20a2a9" containerID="4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56" exitCode=0
Nov 03 23:42:56.453084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.452920    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"
Nov 03 23:42:56.453084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.452948    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc" event={"ID":"c2326428-8e4f-48ed-847b-efdfed20a2a9","Type":"ContainerDied","Data":"4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56"}
Nov 03 23:42:56.453084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.453012    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc" event={"ID":"c2326428-8e4f-48ed-847b-efdfed20a2a9","Type":"ContainerDied","Data":"f07fc6196cf00910e50c6acc1fd65ed3458eb0f21c647f0b5232594497caad4e"}
Nov 03 23:42:56.453084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.453041    5074 scope.go:122] "RemoveContainer" containerID="4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56"
Nov 03 23:42:56.463767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.463749    5074 scope.go:122] "RemoveContainer" containerID="4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56"
Nov 03 23:42:56.464077 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:56.464058    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56\": container with ID starting with 4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56 not found: ID does not exist" containerID="4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56"
Nov 03 23:42:56.464130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.464085    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56"} err="failed to get container status \"4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56\": rpc error: code = NotFound desc = could not find container \"4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56\": container with ID starting with 4f07bb4eac28689be5901ca4511896750972479b883aac20ca6c72023f2c4e56 not found: ID does not exist"
Nov 03 23:42:56.478130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.478105    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"]
Nov 03 23:42:56.484455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:56.484431    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d887c9bbc-m7zcc"]
Nov 03 23:42:57.018820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:42:57.018786    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2326428-8e4f-48ed-847b-efdfed20a2a9" path="/var/lib/kubelet/pods/c2326428-8e4f-48ed-847b-efdfed20a2a9/volumes"
Nov 03 23:42:58.026307 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:42:58.026272    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:43:00.074053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.074012    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"]
Nov 03 23:43:00.074867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.074847    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c2326428-8e4f-48ed-847b-efdfed20a2a9" containerName="payload-processing"
Nov 03 23:43:00.074964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.074869    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2326428-8e4f-48ed-847b-efdfed20a2a9" containerName="payload-processing"
Nov 03 23:43:00.075026 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.075007    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2326428-8e4f-48ed-847b-efdfed20a2a9" containerName="payload-processing"
Nov 03 23:43:00.104389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.104360    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"]
Nov 03 23:43:00.104600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.104489    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.110049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.110019    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:43:00.122953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.122929    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:43:00.181932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.177810    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.181932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.177867    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjjl\" (UniqueName: \"kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.181932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.177983    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.279309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.279271    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.279309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.279310    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjjl\" (UniqueName: \"kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.279549 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.279389    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.280389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.280345    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.282069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.282043    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.293360 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.293332    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjjl\" (UniqueName: \"kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl\") pod \"maas-api-revoke-sub-7f16a77f-rwhpk\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.472244 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.472210    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:00.768474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:00.768445    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"]
Nov 03 23:43:00.769923 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:00.769889    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df2221d_80a4_4bbd_8229_77c51b54f3bc.slice/crio-f9921c42f35ba1dc2b348e344459d9cab269cea4a5f7ddd94709ebacc289ef24 WatchSource:0}: Error finding container f9921c42f35ba1dc2b348e344459d9cab269cea4a5f7ddd94709ebacc289ef24: Status 404 returned error can't find the container with id f9921c42f35ba1dc2b348e344459d9cab269cea4a5f7ddd94709ebacc289ef24
Nov 03 23:43:01.490276 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:01.490168    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk" event={"ID":"4df2221d-80a4-4bbd-8229-77c51b54f3bc","Type":"ContainerStarted","Data":"1c39c64cf6a42e9ca09674815ccf9aa45d6f2c100db0fb5313881b4c9a56384e"}
Nov 03 23:43:01.490276 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:01.490237    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk" event={"ID":"4df2221d-80a4-4bbd-8229-77c51b54f3bc","Type":"ContainerStarted","Data":"f9921c42f35ba1dc2b348e344459d9cab269cea4a5f7ddd94709ebacc289ef24"}
Nov 03 23:43:01.515528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:01.515470    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk" podStartSLOduration=1.515454937 podStartE2EDuration="1.515454937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:01.510727495 +0000 UTC m=+846.610327640" watchObservedRunningTime="2026-11-03 23:43:01.515454937 +0000 UTC m=+846.615055081"
Nov 03 23:43:02.499814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:02.499773    5074 generic.go:356] "Generic (PLEG): container finished" podID="4df2221d-80a4-4bbd-8229-77c51b54f3bc" containerID="1c39c64cf6a42e9ca09674815ccf9aa45d6f2c100db0fb5313881b4c9a56384e" exitCode=0
Nov 03 23:43:02.500313 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:02.499841    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk" event={"ID":"4df2221d-80a4-4bbd-8229-77c51b54f3bc","Type":"ContainerDied","Data":"1c39c64cf6a42e9ca09674815ccf9aa45d6f2c100db0fb5313881b4c9a56384e"}
Nov 03 23:43:03.104864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.104815    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"]
Nov 03 23:43:03.109048 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.109023    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.120575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.120540    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"]
Nov 03 23:43:03.221613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.221580    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.221841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.221633    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ccn\" (UniqueName: \"kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.221841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.221716    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.323068 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.323034    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.323254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.323082    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ccn\" (UniqueName: \"kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.323254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.323145    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.323863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.323827    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.325614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.325585    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.336642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.336619    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ccn\" (UniqueName: \"kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn\") pod \"maas-api-revoke-sub-4d8a9756-mvhcb\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.446083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.446047    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:03.482251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.482221    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:43:03.495874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.495845    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:43:03.552829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.548751    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:43:03.732314 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.732284    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"]
Nov 03 23:43:03.735188 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:03.735157    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715ba7d1_4d00_42b2_a217_604b9963bef8.slice/crio-73b9d861088240b0be21832a942c5ff6aac9d9a8aa7ca7a0b24436ad2cf434af WatchSource:0}: Error finding container 73b9d861088240b0be21832a942c5ff6aac9d9a8aa7ca7a0b24436ad2cf434af: Status 404 returned error can't find the container with id 73b9d861088240b0be21832a942c5ff6aac9d9a8aa7ca7a0b24436ad2cf434af
Nov 03 23:43:03.757778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.757754    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:03.831806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.831775    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca\") pod \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") "
Nov 03 23:43:03.831989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.831868    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca\") pod \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") "
Nov 03 23:43:03.832057 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.831999    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl\" (UniqueName: \"kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl\") pod \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\" (UID: \"4df2221d-80a4-4bbd-8229-77c51b54f3bc\") "
Nov 03 23:43:03.832181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.832156    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca" pod "4df2221d-80a4-4bbd-8229-77c51b54f3bc" (UID: "4df2221d-80a4-4bbd-8229-77c51b54f3bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:03.832331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.832315    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:03.834301 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.834267    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca" pod "4df2221d-80a4-4bbd-8229-77c51b54f3bc" (UID: "4df2221d-80a4-4bbd-8229-77c51b54f3bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:03.834697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.834675    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl" pod "4df2221d-80a4-4bbd-8229-77c51b54f3bc" (UID: "4df2221d-80a4-4bbd-8229-77c51b54f3bc"). InnerVolumeSpecName "kube-api-access-hqjjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:03.933863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.933829    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df2221d-80a4-4bbd-8229-77c51b54f3bc-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:03.933863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:03.933865    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjjl\" (UniqueName: \"kubernetes.io/projected/4df2221d-80a4-4bbd-8229-77c51b54f3bc-kube-api-access-hqjjl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:04.519757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:04.518798    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"
Nov 03 23:43:04.519757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:04.518830    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk" event={"ID":"4df2221d-80a4-4bbd-8229-77c51b54f3bc","Type":"ContainerDied","Data":"f9921c42f35ba1dc2b348e344459d9cab269cea4a5f7ddd94709ebacc289ef24"}
Nov 03 23:43:04.519757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:04.518891    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9921c42f35ba1dc2b348e344459d9cab269cea4a5f7ddd94709ebacc289ef24"
Nov 03 23:43:04.521035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:04.521009    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb" event={"ID":"715ba7d1-4d00-42b2-a217-604b9963bef8","Type":"ContainerStarted","Data":"c12945f478e29842baaaea63583fa12b38b2de825a1e6af75aa448f591450b11"}
Nov 03 23:43:04.521156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:04.521061    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb" event={"ID":"715ba7d1-4d00-42b2-a217-604b9963bef8","Type":"ContainerStarted","Data":"73b9d861088240b0be21832a942c5ff6aac9d9a8aa7ca7a0b24436ad2cf434af"}
Nov 03 23:43:04.544451 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:04.544404    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb" podStartSLOduration=1.54439401 podStartE2EDuration="1.54439401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:04.542985311 +0000 UTC m=+849.642585456" watchObservedRunningTime="2026-11-03 23:43:04.54439401 +0000 UTC m=+849.643994174"
Nov 03 23:43:05.529165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:05.529129    5074 generic.go:356] "Generic (PLEG): container finished" podID="715ba7d1-4d00-42b2-a217-604b9963bef8" containerID="c12945f478e29842baaaea63583fa12b38b2de825a1e6af75aa448f591450b11" exitCode=0
Nov 03 23:43:05.529554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:05.529200    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb" event={"ID":"715ba7d1-4d00-42b2-a217-604b9963bef8","Type":"ContainerDied","Data":"c12945f478e29842baaaea63583fa12b38b2de825a1e6af75aa448f591450b11"}
Nov 03 23:43:06.688115 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.688089    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:06.766048 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.766014    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca\") pod \"715ba7d1-4d00-42b2-a217-604b9963bef8\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") "
Nov 03 23:43:06.766207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.766109    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca\") pod \"715ba7d1-4d00-42b2-a217-604b9963bef8\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") "
Nov 03 23:43:06.766207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.766164    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn\" (UniqueName: \"kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn\") pod \"715ba7d1-4d00-42b2-a217-604b9963bef8\" (UID: \"715ba7d1-4d00-42b2-a217-604b9963bef8\") "
Nov 03 23:43:06.766479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.766450    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca" pod "715ba7d1-4d00-42b2-a217-604b9963bef8" (UID: "715ba7d1-4d00-42b2-a217-604b9963bef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:06.768203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.768179    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca" pod "715ba7d1-4d00-42b2-a217-604b9963bef8" (UID: "715ba7d1-4d00-42b2-a217-604b9963bef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:06.768370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.768348    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn" pod "715ba7d1-4d00-42b2-a217-604b9963bef8" (UID: "715ba7d1-4d00-42b2-a217-604b9963bef8"). InnerVolumeSpecName "kube-api-access-c8ccn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:06.867162 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.867092    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8ccn\" (UniqueName: \"kubernetes.io/projected/715ba7d1-4d00-42b2-a217-604b9963bef8-kube-api-access-c8ccn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:06.867162 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.867120    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:06.867162 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:06.867130    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715ba7d1-4d00-42b2-a217-604b9963bef8-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:07.543384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.543352    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"
Nov 03 23:43:07.543564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.543390    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb" event={"ID":"715ba7d1-4d00-42b2-a217-604b9963bef8","Type":"ContainerDied","Data":"73b9d861088240b0be21832a942c5ff6aac9d9a8aa7ca7a0b24436ad2cf434af"}
Nov 03 23:43:07.543564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.543445    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b9d861088240b0be21832a942c5ff6aac9d9a8aa7ca7a0b24436ad2cf434af"
Nov 03 23:43:07.698566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.698533    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"]
Nov 03 23:43:07.699268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.699246    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4df2221d-80a4-4bbd-8229-77c51b54f3bc" containerName="revoke-keys"
Nov 03 23:43:07.699379 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.699270    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df2221d-80a4-4bbd-8229-77c51b54f3bc" containerName="revoke-keys"
Nov 03 23:43:07.699379 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.699319    5074 cpu_manager.go:414] "containerMap: removing container" podUID="715ba7d1-4d00-42b2-a217-604b9963bef8" containerName="revoke-keys"
Nov 03 23:43:07.699379 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.699328    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715ba7d1-4d00-42b2-a217-604b9963bef8" containerName="revoke-keys"
Nov 03 23:43:07.699537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.699457    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="715ba7d1-4d00-42b2-a217-604b9963bef8" containerName="revoke-keys"
Nov 03 23:43:07.699537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.699479    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df2221d-80a4-4bbd-8229-77c51b54f3bc" containerName="revoke-keys"
Nov 03 23:43:07.703000 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.702980    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.710418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.710392    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"openshift-service-ca.crt\""
Nov 03 23:43:07.710418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.710405    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"kube-root-ca.crt\""
Nov 03 23:43:07.710629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.710559    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"default-dockercfg-fjmq9\""
Nov 03 23:43:07.710721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.710706    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"distinct-2-w3-c60893-kserve-self-signed-certs\""
Nov 03 23:43:07.714476 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.714243    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"]
Nov 03 23:43:07.879575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.879474    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.879575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.879545    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.879818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.879679    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.879884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.879860    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.879941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.879917    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.880029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.880012    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrgh2\" (UniqueName: \"kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981177    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981307    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981382    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981419    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrgh2\" (UniqueName: \"kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981532    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981578    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981641    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981916    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.981943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.981941    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.984436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.984410    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.985035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.985013    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:07.995584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:07.995550    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrgh2\" (UniqueName: \"kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2\") pod \"distinct-2-w3-c60893-kserve-5b788b848d-t6nxw\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:08.024823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:08.024800    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:08.313094 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:08.313067    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"]
Nov 03 23:43:08.313599 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:08.313572    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5a65ffa_55ad_47f5_a293_9d2ac3ae27a3.slice/crio-52281e04e90bf8b90440191a682636e152f2ab9dd8e9644502c2ca04e1ca82b4 WatchSource:0}: Error finding container 52281e04e90bf8b90440191a682636e152f2ab9dd8e9644502c2ca04e1ca82b4: Status 404 returned error can't find the container with id 52281e04e90bf8b90440191a682636e152f2ab9dd8e9644502c2ca04e1ca82b4
Nov 03 23:43:08.561984 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:08.561881    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" event={"ID":"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3","Type":"ContainerStarted","Data":"ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56"}
Nov 03 23:43:08.561984 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:08.561944    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" event={"ID":"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3","Type":"ContainerStarted","Data":"52281e04e90bf8b90440191a682636e152f2ab9dd8e9644502c2ca04e1ca82b4"}
Nov 03 23:43:08.585203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:08.585155    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podStartSLOduration=1.585140744 podStartE2EDuration="1.585140744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:08.584276307 +0000 UTC m=+853.683876453" watchObservedRunningTime="2026-11-03 23:43:08.585140744 +0000 UTC m=+853.684740888"
Nov 03 23:43:09.025365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:09.025330    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:11.032534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:11.032484    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"]
Nov 03 23:43:13.455394 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:13.454869    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"]
Nov 03 23:43:15.371273 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.371233    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"]
Nov 03 23:43:15.375525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.375503    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.380513 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380484    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.380654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380533    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84g2\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.380654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380608    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.380795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380764    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.380926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380830    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.380976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380958    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.381009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.380991    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.381050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.381029    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.381114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.381101    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.389064 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.389038    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c60893-openshift-default-dockercfg-hwjgp\""
Nov 03 23:43:15.392187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.392164    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"]
Nov 03 23:43:15.482264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482217    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482273    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482310    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482329    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482357    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482379    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84g2\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482398    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482457    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482505    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.482914 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482752    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.483004 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482951    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.483004 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.482975    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.483128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.483108    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.483355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.483332    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.484911 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.484888    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.485003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.484891    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.505983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.505960    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.506242 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.506213    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84g2\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.692488 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.692449    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:15.980715 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.980685    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"]
Nov 03 23:43:15.981631 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:15.981607    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cc8da7_16d5_41d0_93b7_b34ad0dc0f33.slice/crio-208200bc0ab1941358932465fb2c8c21410c1562566fcbf0ef69c93264fe032f WatchSource:0}: Error finding container 208200bc0ab1941358932465fb2c8c21410c1562566fcbf0ef69c93264fe032f: Status 404 returned error can't find the container with id 208200bc0ab1941358932465fb2c8c21410c1562566fcbf0ef69c93264fe032f
Nov 03 23:43:15.984008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.983976    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:15.984078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.984042    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:15.984078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:15.984072    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:16.033165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.032646    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"]
Nov 03 23:43:16.034016 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.033984    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx" podUID="486182f8-6936-4ea5-81e6-b21f6b8b060c" containerName="payload-processing" containerID="cri-o://a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b" gracePeriod=25
Nov 03 23:43:16.038134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.038111    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.044089 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.044066    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:43:16.045515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.045495    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:43:16.048386 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.048365    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"]
Nov 03 23:43:16.089692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.089659    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjfs\" (UniqueName: \"kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.089853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.089798    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.089853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.089829    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.191381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.191350    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.191381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.191391    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.191626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.191568    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjfs\" (UniqueName: \"kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.192296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.192272    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.194243 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.194219    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.205769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.205696    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjfs\" (UniqueName: \"kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs\") pod \"maas-api-revoke-sub-6526719a-q247p\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.308998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.308966    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:43:16.363035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.363001    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:16.394154 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.393539    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc\") pod \"486182f8-6936-4ea5-81e6-b21f6b8b060c\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") "
Nov 03 23:43:16.394154 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.393632    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume\") pod \"486182f8-6936-4ea5-81e6-b21f6b8b060c\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") "
Nov 03 23:43:16.394154 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.393687    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials\") pod \"486182f8-6936-4ea5-81e6-b21f6b8b060c\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") "
Nov 03 23:43:16.394154 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.393726    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume\") pod \"486182f8-6936-4ea5-81e6-b21f6b8b060c\" (UID: \"486182f8-6936-4ea5-81e6-b21f6b8b060c\") "
Nov 03 23:43:16.394154 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.394115    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume" pod "486182f8-6936-4ea5-81e6-b21f6b8b060c" (UID: "486182f8-6936-4ea5-81e6-b21f6b8b060c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:16.394812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.394367    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:16.394812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.394399    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume" pod "486182f8-6936-4ea5-81e6-b21f6b8b060c" (UID: "486182f8-6936-4ea5-81e6-b21f6b8b060c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:16.396670 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.396643    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc" pod "486182f8-6936-4ea5-81e6-b21f6b8b060c" (UID: "486182f8-6936-4ea5-81e6-b21f6b8b060c"). InnerVolumeSpecName "kube-api-access-qxvmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:16.396804 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.396693    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials" pod "486182f8-6936-4ea5-81e6-b21f6b8b060c" (UID: "486182f8-6936-4ea5-81e6-b21f6b8b060c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:16.500777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.495734    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:16.500777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.495783    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486182f8-6936-4ea5-81e6-b21f6b8b060c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:16.500777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.495794    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvmc\" (UniqueName: \"kubernetes.io/projected/486182f8-6936-4ea5-81e6-b21f6b8b060c-kube-api-access-qxvmc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:16.629017 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.628923    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" event={"ID":"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33","Type":"ContainerStarted","Data":"db72f39c672b78c740140110f7129a0589f2e4365e05973dc587dc2caf90c872"}
Nov 03 23:43:16.629293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.629267    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" event={"ID":"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33","Type":"ContainerStarted","Data":"208200bc0ab1941358932465fb2c8c21410c1562566fcbf0ef69c93264fe032f"}
Nov 03 23:43:16.630590 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.630562    5074 generic.go:356] "Generic (PLEG): container finished" podID="486182f8-6936-4ea5-81e6-b21f6b8b060c" containerID="a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b" exitCode=0
Nov 03 23:43:16.630694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.630609    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"
Nov 03 23:43:16.630694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.630665    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx" event={"ID":"486182f8-6936-4ea5-81e6-b21f6b8b060c","Type":"ContainerDied","Data":"a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b"}
Nov 03 23:43:16.630889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.630722    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx" event={"ID":"486182f8-6936-4ea5-81e6-b21f6b8b060c","Type":"ContainerDied","Data":"fe1904516d828da3eb0a1040a069b3ffc4b66837c7af6a4b8bfb4f47ee9eb5fa"}
Nov 03 23:43:16.630889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.630777    5074 scope.go:122] "RemoveContainer" containerID="a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b"
Nov 03 23:43:16.651760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.647980    5074 scope.go:122] "RemoveContainer" containerID="a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b"
Nov 03 23:43:16.651760 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:43:16.648317    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b\": container with ID starting with a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b not found: ID does not exist" containerID="a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b"
Nov 03 23:43:16.651760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.648352    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b"} err="failed to get container status \"a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b\": rpc error: code = NotFound desc = could not find container \"a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b\": container with ID starting with a41896190e97d34f23e43a6e98e6df42ace983f91cb87342a87cc19ded34e47b not found: ID does not exist"
Nov 03 23:43:16.665003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.664946    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podStartSLOduration=1.664926687 podStartE2EDuration="1.664926687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:16.661909907 +0000 UTC m=+861.761510051" watchObservedRunningTime="2026-11-03 23:43:16.664926687 +0000 UTC m=+861.764526833"
Nov 03 23:43:16.670350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.670323    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"]
Nov 03 23:43:16.674763 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:16.671838    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b56aa20_a5ae_42b9_b82d_d886c938c2f8.slice/crio-2282960da20c1d212b16cafd65eeab30ba9eac73d6af82cf46542dea3ad991e7 WatchSource:0}: Error finding container 2282960da20c1d212b16cafd65eeab30ba9eac73d6af82cf46542dea3ad991e7: Status 404 returned error can't find the container with id 2282960da20c1d212b16cafd65eeab30ba9eac73d6af82cf46542dea3ad991e7
Nov 03 23:43:16.687997 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.687937    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"]
Nov 03 23:43:16.695973 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.692092    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b4ff889d7-p4ndx"]
Nov 03 23:43:16.707411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:16.697071    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:17.039282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:17.039248    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486182f8-6936-4ea5-81e6-b21f6b8b060c" path="/var/lib/kubelet/pods/486182f8-6936-4ea5-81e6-b21f6b8b060c/volumes"
Nov 03 23:43:17.645068 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:17.645023    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p" event={"ID":"6b56aa20-a5ae-42b9-b82d-d886c938c2f8","Type":"ContainerStarted","Data":"63d5d7bab08ce773d6a5fbcb4f78e76a8555aeeea61892e645962e758ecbde9a"}
Nov 03 23:43:17.645654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:17.645093    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p" event={"ID":"6b56aa20-a5ae-42b9-b82d-d886c938c2f8","Type":"ContainerStarted","Data":"2282960da20c1d212b16cafd65eeab30ba9eac73d6af82cf46542dea3ad991e7"}
Nov 03 23:43:17.671513 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:17.671456    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p" podStartSLOduration=1.6714420890000001 podStartE2EDuration="1.671442089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:17.669252923 +0000 UTC m=+862.768853068" watchObservedRunningTime="2026-11-03 23:43:17.671442089 +0000 UTC m=+862.771042241"
Nov 03 23:43:17.695215 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:17.695182    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 23:43:17.695374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:17.695231    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 23:43:18.034765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.028105    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:18.061434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.061405    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:18.656179 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.656146    5074 generic.go:356] "Generic (PLEG): container finished" podID="6b56aa20-a5ae-42b9-b82d-d886c938c2f8" containerID="63d5d7bab08ce773d6a5fbcb4f78e76a8555aeeea61892e645962e758ecbde9a" exitCode=0
Nov 03 23:43:18.656645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.656236    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p" event={"ID":"6b56aa20-a5ae-42b9-b82d-d886c938c2f8","Type":"ContainerDied","Data":"63d5d7bab08ce773d6a5fbcb4f78e76a8555aeeea61892e645962e758ecbde9a"}
Nov 03 23:43:18.668581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.668555    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:43:18.692894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.692868    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 23:43:18.693061 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:18.692917    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 23:43:19.076920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.076879    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"]
Nov 03 23:43:19.077474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.077455    5074 cpu_manager.go:414] "containerMap: removing container" podUID="486182f8-6936-4ea5-81e6-b21f6b8b060c" containerName="payload-processing"
Nov 03 23:43:19.077580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.077477    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486182f8-6936-4ea5-81e6-b21f6b8b060c" containerName="payload-processing"
Nov 03 23:43:19.077665 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.077652    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="486182f8-6936-4ea5-81e6-b21f6b8b060c" containerName="payload-processing"
Nov 03 23:43:19.081291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.081269    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.142663 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.140964    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"]
Nov 03 23:43:19.151968 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.151932    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.151984    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnsc\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152121    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152175    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152247    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152295    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152326    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152356    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.152479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.152391    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253157    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253208    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253231    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253257    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253287    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253334    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253372    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnsc\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253447    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253494    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253635    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253837    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.253982 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.253928    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.254116 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.254085    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.254223 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.254175    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.255795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.255776    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.256063 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.256045    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.267067 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.267042    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnsc\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.267546 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.267529    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.397334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.397242    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:19.693539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.693501    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 23:43:19.694021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.693549    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 23:43:19.751497 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.751465    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"]
Nov 03 23:43:19.752298 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:19.752270    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3eb966_8254_46ab_ac6d_e609f11b651c.slice/crio-6cafdb449ec3aa63c7834c1488948b2f6044dd6472c4d30f5102a79fde73131f WatchSource:0}: Error finding container 6cafdb449ec3aa63c7834c1488948b2f6044dd6472c4d30f5102a79fde73131f: Status 404 returned error can't find the container with id 6cafdb449ec3aa63c7834c1488948b2f6044dd6472c4d30f5102a79fde73131f
Nov 03 23:43:19.755412 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.755380    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:19.755509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.755443    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:19.755509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.755475    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:19.913766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.909931    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:19.965770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.962277    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs\" (UniqueName: \"kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs\") pod \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") "
Nov 03 23:43:19.965770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.962336    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca\") pod \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") "
Nov 03 23:43:19.965770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.962410    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca\") pod \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\" (UID: \"6b56aa20-a5ae-42b9-b82d-d886c938c2f8\") "
Nov 03 23:43:19.965770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.963135    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca" pod "6b56aa20-a5ae-42b9-b82d-d886c938c2f8" (UID: "6b56aa20-a5ae-42b9-b82d-d886c938c2f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:19.965770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.965330    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs" pod "6b56aa20-a5ae-42b9-b82d-d886c938c2f8" (UID: "6b56aa20-a5ae-42b9-b82d-d886c938c2f8"). InnerVolumeSpecName "kube-api-access-2wjfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:19.965770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:19.965522    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca" pod "6b56aa20-a5ae-42b9-b82d-d886c938c2f8" (UID: "6b56aa20-a5ae-42b9-b82d-d886c938c2f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:20.064340 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.064302    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:20.064340 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.064339    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjfs\" (UniqueName: \"kubernetes.io/projected/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-kube-api-access-2wjfs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:20.064621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.064355    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b56aa20-a5ae-42b9-b82d-d886c938c2f8-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:20.681496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.681415    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" event={"ID":"1f3eb966-8254-46ab-ac6d-e609f11b651c","Type":"ContainerStarted","Data":"2df3083dfa6fe33e5e73100dd50035c13c6da7b3a3f07612df1702603ac1022a"}
Nov 03 23:43:20.681496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.681469    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" event={"ID":"1f3eb966-8254-46ab-ac6d-e609f11b651c","Type":"ContainerStarted","Data":"6cafdb449ec3aa63c7834c1488948b2f6044dd6472c4d30f5102a79fde73131f"}
Nov 03 23:43:20.683260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.683236    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"
Nov 03 23:43:20.683260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.683244    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p" event={"ID":"6b56aa20-a5ae-42b9-b82d-d886c938c2f8","Type":"ContainerDied","Data":"2282960da20c1d212b16cafd65eeab30ba9eac73d6af82cf46542dea3ad991e7"}
Nov 03 23:43:20.683470 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.683285    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2282960da20c1d212b16cafd65eeab30ba9eac73d6af82cf46542dea3ad991e7"
Nov 03 23:43:20.693514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.693493    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 23:43:20.693625 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.693543    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 23:43:20.746761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:20.746612    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podStartSLOduration=1.746592666 podStartE2EDuration="1.746592666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:20.744322866 +0000 UTC m=+865.843923011" watchObservedRunningTime="2026-11-03 23:43:20.746592666 +0000 UTC m=+865.846192814"
Nov 03 23:43:21.402762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:21.402234    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:21.410763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:21.408859    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:43:21.410763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:21.408925    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 23:43:21.711656 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:21.711621    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 23:43:21.712291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:21.712245    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 23:43:22.402282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.402245    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:43:22.402601 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.402553    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 23:43:22.693084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.693052    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 23:43:22.693284 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.693107    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 23:43:22.793361 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.793323    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"]
Nov 03 23:43:22.793937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.793920    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6b56aa20-a5ae-42b9-b82d-d886c938c2f8" containerName="revoke-keys"
Nov 03 23:43:22.794047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.793940    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b56aa20-a5ae-42b9-b82d-d886c938c2f8" containerName="revoke-keys"
Nov 03 23:43:22.794083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.794057    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b56aa20-a5ae-42b9-b82d-d886c938c2f8" containerName="revoke-keys"
Nov 03 23:43:22.797789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.797768    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.926762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926498    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.926762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926548    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.926762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926590    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.926762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926682    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.926762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926722    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.927189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926814    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.927189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926844    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mjh\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.927189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.926886    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.927189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.927038    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:22.938439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:22.938406    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"]
Nov 03 23:43:23.032865 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.032733    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.033629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.033581    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.033852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.033825    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.034510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.034482    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.034725 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.034705    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.034915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.034888    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.039904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.035138    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040027 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.039933    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mjh\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040027 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.039967    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.036283    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.037399    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.039854    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.034189    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.035513    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.040495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.040438    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.041424 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.041299    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.058199 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.057120    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mjh\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.061996 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.061972    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.136193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.136157    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:23.398258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.398157    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:43:23.398258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.398225    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 23:43:23.473813 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:23.473757    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21434b03_716d_4849_b8c3_490b5f807b51.slice/crio-ccde984e54eab29ec37a97fa7efce2689769fc98294b93b3a6650554b7bd7186 WatchSource:0}: Error finding container ccde984e54eab29ec37a97fa7efce2689769fc98294b93b3a6650554b7bd7186: Status 404 returned error can't find the container with id ccde984e54eab29ec37a97fa7efce2689769fc98294b93b3a6650554b7bd7186
Nov 03 23:43:23.474845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.474816    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"]
Nov 03 23:43:23.476978 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.476940    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:23.477092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.477028    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:23.477092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.477074    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:23.772819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.771076    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" event={"ID":"21434b03-716d-4849-b8c3-490b5f807b51","Type":"ContainerStarted","Data":"8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817"}
Nov 03 23:43:23.772819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:23.771145    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" event={"ID":"21434b03-716d-4849-b8c3-490b5f807b51","Type":"ContainerStarted","Data":"ccde984e54eab29ec37a97fa7efce2689769fc98294b93b3a6650554b7bd7186"}
Nov 03 23:43:24.149348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.144901    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:24.150070 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.150035    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:24.150311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.150269    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:24.166154 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.166118    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:24.180915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.167921    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:24.182547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.182522    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 03 23:43:24.214822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.214770    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podStartSLOduration=2.214752253 podStartE2EDuration="2.214752253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:23.831632359 +0000 UTC m=+868.931232504" watchObservedRunningTime="2026-11-03 23:43:24.214752253 +0000 UTC m=+869.314352395"
Nov 03 23:43:24.401828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.400855    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:43:24.401828 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:24.400929    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 23:43:25.138206 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:25.138163    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:25.138553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:25.138512    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:25.407599 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:25.407505    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:43:25.408238 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:25.408198    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 23:43:26.138764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:26.138108    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:26.138764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:26.138170    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:26.406812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:26.402121    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:43:26.406812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:26.402182    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 23:43:27.139766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:27.137085    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:27.139766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:27.137163    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:27.691718 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:27.691684    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:27.692726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:27.692694    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:27.693697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:27.693676    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 03 23:43:28.059416 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:43:28.059331    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:43:28.137186 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.137151    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:28.137359 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.137206    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:28.327817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.324599    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"]
Nov 03 23:43:28.342141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.342118    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"]
Nov 03 23:43:28.342368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.342348    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.348155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.348126    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:43:28.348297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.348187    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:43:28.373157 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.373129    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bf77\" (UniqueName: \"kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.373295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.373276    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.373344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.373319    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.474132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.474092    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.474304 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.474155    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.474304 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.474278    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bf77\" (UniqueName: \"kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.474883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.474853    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.476420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.476400    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.480766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.480697    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" podUID="9e26b7c8-1dee-44f5-971d-9531dfdda203" containerName="main" containerID="cri-o://261a48454147fc538ad45e0b756dc54a533c4390914a2d92475d89ca62491daf" gracePeriod=45
Nov 03 23:43:28.487326 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.487307    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bf77\" (UniqueName: \"kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77\") pod \"maas-api-revoke-sub-f46b7b73-jgkfk\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.521388 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.516324    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" podUID="9e26b7c8-1dee-44f5-971d-9531dfdda203" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": dial tcp 10.132.0.89:8000: connect: connection refused"
Nov 03 23:43:28.671664 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.671635    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:28.867281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.866471    5074 generic.go:356] "Generic (PLEG): container finished" podID="9e26b7c8-1dee-44f5-971d-9531dfdda203" containerID="261a48454147fc538ad45e0b756dc54a533c4390914a2d92475d89ca62491daf" exitCode=0
Nov 03 23:43:28.867281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.866832    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" event={"ID":"9e26b7c8-1dee-44f5-971d-9531dfdda203","Type":"ContainerDied","Data":"261a48454147fc538ad45e0b756dc54a533c4390914a2d92475d89ca62491daf"}
Nov 03 23:43:28.911763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:28.898034    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.004404    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache\") pod \"9e26b7c8-1dee-44f5-971d-9531dfdda203\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") "
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.004505    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs\") pod \"9e26b7c8-1dee-44f5-971d-9531dfdda203\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") "
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.004563    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home\") pod \"9e26b7c8-1dee-44f5-971d-9531dfdda203\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") "
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.004627    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir\") pod \"9e26b7c8-1dee-44f5-971d-9531dfdda203\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") "
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.004717    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4\" (UniqueName: \"kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4\") pod \"9e26b7c8-1dee-44f5-971d-9531dfdda203\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") "
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.004834    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm\") pod \"9e26b7c8-1dee-44f5-971d-9531dfdda203\" (UID: \"9e26b7c8-1dee-44f5-971d-9531dfdda203\") "
Nov 03 23:43:29.020762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.019835    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache" pod "9e26b7c8-1dee-44f5-971d-9531dfdda203" (UID: "9e26b7c8-1dee-44f5-971d-9531dfdda203"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:29.043184 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.042888    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home" pod "9e26b7c8-1dee-44f5-971d-9531dfdda203" (UID: "9e26b7c8-1dee-44f5-971d-9531dfdda203"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:29.043966 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.043925    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir" pod "9e26b7c8-1dee-44f5-971d-9531dfdda203" (UID: "9e26b7c8-1dee-44f5-971d-9531dfdda203"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:29.043966 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.043948    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm" pod "9e26b7c8-1dee-44f5-971d-9531dfdda203" (UID: "9e26b7c8-1dee-44f5-971d-9531dfdda203"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:29.051417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.049916    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4" pod "9e26b7c8-1dee-44f5-971d-9531dfdda203" (UID: "9e26b7c8-1dee-44f5-971d-9531dfdda203"). InnerVolumeSpecName "kube-api-access-pp8n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:29.052138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.051760    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs" pod "9e26b7c8-1dee-44f5-971d-9531dfdda203" (UID: "9e26b7c8-1dee-44f5-971d-9531dfdda203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:29.128846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.127223    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:29.128846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.127259    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:29.128846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.127274    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e26b7c8-1dee-44f5-971d-9531dfdda203-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:29.128846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.127287    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:29.128846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.127302    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e26b7c8-1dee-44f5-971d-9531dfdda203-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:29.128846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.127315    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8n4\" (UniqueName: \"kubernetes.io/projected/9e26b7c8-1dee-44f5-971d-9531dfdda203-kube-api-access-pp8n4\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:29.134492 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.134460    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"]
Nov 03 23:43:29.136945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.136921    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:29.137109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.136969    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:29.883001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.882852    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk" event={"ID":"cf8165cf-bc21-4c4c-9126-450c99696f6b","Type":"ContainerStarted","Data":"740a265fb7ff441d105883749f073872314f10d18b0661aeb9f2221f5a49f95c"}
Nov 03 23:43:29.883001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.882949    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk" event={"ID":"cf8165cf-bc21-4c4c-9126-450c99696f6b","Type":"ContainerStarted","Data":"cd8a1c1b426ae3540b3fc3651ab937a60bb1ee6cbb4e2e2d627a999f76f011a2"}
Nov 03 23:43:29.903945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.899698    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n" event={"ID":"9e26b7c8-1dee-44f5-971d-9531dfdda203","Type":"ContainerDied","Data":"7d68ed734463c8791eeb2ada32c4deba37ff26bbd00975766b7b85671d7f3add"}
Nov 03 23:43:29.903945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.899766    5074 scope.go:122] "RemoveContainer" containerID="261a48454147fc538ad45e0b756dc54a533c4390914a2d92475d89ca62491daf"
Nov 03 23:43:29.903945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.899919    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"
Nov 03 23:43:29.919457 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.919393    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk" podStartSLOduration=1.919373561 podStartE2EDuration="1.919373561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:29.91257941 +0000 UTC m=+875.012179555" watchObservedRunningTime="2026-11-03 23:43:29.919373561 +0000 UTC m=+875.018973708"
Nov 03 23:43:29.936763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.936705    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"]
Nov 03 23:43:29.942051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:29.942012    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-b-86358670-kserve-6b6f5c77c6-z6s5n"]
Nov 03 23:43:30.141060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:30.140514    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:30.141060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:30.140573    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:30.963770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:30.963703    5074 generic.go:356] "Generic (PLEG): container finished" podID="cf8165cf-bc21-4c4c-9126-450c99696f6b" containerID="740a265fb7ff441d105883749f073872314f10d18b0661aeb9f2221f5a49f95c" exitCode=0
Nov 03 23:43:30.964253 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:30.963950    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk" event={"ID":"cf8165cf-bc21-4c4c-9126-450c99696f6b","Type":"ContainerDied","Data":"740a265fb7ff441d105883749f073872314f10d18b0661aeb9f2221f5a49f95c"}
Nov 03 23:43:31.064766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:31.063937    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e26b7c8-1dee-44f5-971d-9531dfdda203" path="/var/lib/kubelet/pods/9e26b7c8-1dee-44f5-971d-9531dfdda203/volumes"
Nov 03 23:43:31.156574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:31.156503    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 23:43:31.156874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:31.156814    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 23:43:32.553467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.528861    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"]
Nov 03 23:43:32.553467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.529686    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9e26b7c8-1dee-44f5-971d-9531dfdda203" containerName="main"
Nov 03 23:43:32.553467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.529703    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e26b7c8-1dee-44f5-971d-9531dfdda203" containerName="main"
Nov 03 23:43:32.553467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.529908    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e26b7c8-1dee-44f5-971d-9531dfdda203" containerName="main"
Nov 03 23:43:32.560189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.557332    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.568071 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.568030    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"kube-root-ca.crt\""
Nov 03 23:43:32.568220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.568138    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"default-dockercfg-6wx2m\""
Nov 03 23:43:32.568220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.568214    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"distinct-w4-80f8ba-kserve-self-signed-certs\""
Nov 03 23:43:32.574447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.568333    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"openshift-service-ca.crt\""
Nov 03 23:43:32.627397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.627360    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"]
Nov 03 23:43:32.810819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.803350    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.810819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.803439    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.810819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.803498    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.810819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.803541    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.810819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.803594    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.810819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.803644    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcqv\" (UniqueName: \"kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.930659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.930183    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.930659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.930272    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.930659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.930322    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.930659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.930388    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.930659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.930450    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcqv\" (UniqueName: \"kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.930659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.930605    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.948014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.946878    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.948014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.947158    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.948297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.948267    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.967837 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.964171    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:32.967837 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:32.965852    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:33.089846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.083639    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:33.180696 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.178827    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca\") pod \"cf8165cf-bc21-4c4c-9126-450c99696f6b\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") "
Nov 03 23:43:33.180696 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.178945    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca\") pod \"cf8165cf-bc21-4c4c-9126-450c99696f6b\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") "
Nov 03 23:43:33.180696 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.179028    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77\" (UniqueName: \"kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77\") pod \"cf8165cf-bc21-4c4c-9126-450c99696f6b\" (UID: \"cf8165cf-bc21-4c4c-9126-450c99696f6b\") "
Nov 03 23:43:33.191009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.189453    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca" pod "cf8165cf-bc21-4c4c-9126-450c99696f6b" (UID: "cf8165cf-bc21-4c4c-9126-450c99696f6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:33.193985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.193126    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcqv\" (UniqueName: \"kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv\") pod \"distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:33.263648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.263608    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca" pod "cf8165cf-bc21-4c4c-9126-450c99696f6b" (UID: "cf8165cf-bc21-4c4c-9126-450c99696f6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:33.266999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.266973    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"
Nov 03 23:43:33.267704 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.267688    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:33.285760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.281972    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:43:33.285760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.282060    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.285760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.285201    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:33.285760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.285234    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8165cf-bc21-4c4c-9126-450c99696f6b-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:33.369778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.368424    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77" pod "cf8165cf-bc21-4c4c-9126-450c99696f6b" (UID: "cf8165cf-bc21-4c4c-9126-450c99696f6b"). InnerVolumeSpecName "kube-api-access-2bf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:33.419684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.419604    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bf77\" (UniqueName: \"kubernetes.io/projected/cf8165cf-bc21-4c4c-9126-450c99696f6b-kube-api-access-2bf77\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:33.464359 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.464325    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"]
Nov 03 23:43:33.465483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.465440    5074 cpu_manager.go:414] "containerMap: removing container" podUID="cf8165cf-bc21-4c4c-9126-450c99696f6b" containerName="revoke-keys"
Nov 03 23:43:33.465660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.465643    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8165cf-bc21-4c4c-9126-450c99696f6b" containerName="revoke-keys"
Nov 03 23:43:33.465976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.465958    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8165cf-bc21-4c4c-9126-450c99696f6b" containerName="revoke-keys"
Nov 03 23:43:33.530592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.530436    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"]
Nov 03 23:43:33.530592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.530491    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk" event={"ID":"cf8165cf-bc21-4c4c-9126-450c99696f6b","Type":"ContainerDied","Data":"cd8a1c1b426ae3540b3fc3651ab937a60bb1ee6cbb4e2e2d627a999f76f011a2"}
Nov 03 23:43:33.530592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.530529    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8a1c1b426ae3540b3fc3651ab937a60bb1ee6cbb4e2e2d627a999f76f011a2"
Nov 03 23:43:33.530890 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.530624    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.663757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.663355    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.663757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.663468    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.663757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.663548    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ggr\" (UniqueName: \"kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.781580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.779427    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ggr\" (UniqueName: \"kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.781580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.779591    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.781580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.779684    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.792894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.792760    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.795313 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.795223    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:33.834761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:33.826522    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ggr\" (UniqueName: \"kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr\") pod \"maas-api-revoke-sub-46a36547-ldsjw\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:34.032764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.032472    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:34.141874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.141330    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:43:34.141874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.141399    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:34.333308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.332823    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:43:34.333308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.332886    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:34.358602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.358571    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:34.359532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.359509    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:34.360260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.360092    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 03 23:43:34.363692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.363674    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"]
Nov 03 23:43:34.367214 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:34.367190    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc631f8d7_c832_4e70_b291_4e8629e024a5.slice/crio-0dae702945e4d67df117d141c91d3c75d5e40712971151150843a90493301991 WatchSource:0}: Error finding container 0dae702945e4d67df117d141c91d3c75d5e40712971151150843a90493301991: Status 404 returned error can't find the container with id 0dae702945e4d67df117d141c91d3c75d5e40712971151150843a90493301991
Nov 03 23:43:34.515887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.515389    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"]
Nov 03 23:43:34.894544 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.894510    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"]
Nov 03 23:43:34.921071 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.921034    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"]
Nov 03 23:43:34.921254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.921182    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:34.932582 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:34.932510    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"distinct-w1-5cb349-kserve-self-signed-certs\""
Nov 03 23:43:35.029241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.029209    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.029592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.029568    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.029765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.029729    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.030144 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.030122    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.030311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.030292    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85nn\" (UniqueName: \"kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.030499 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.030476    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.132368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.132274    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.132368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.132326    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.132596 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.132439    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.132596 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.132493    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85nn\" (UniqueName: \"kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.132716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.132652    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.132716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.132706    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.133232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.133202    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.134642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.134614    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.135229 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.135204    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.151676 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.151504    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.152302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.151916    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.156125 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.156106    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85nn\" (UniqueName: \"kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn\") pod \"distinct-w1-5cb349-kserve-545c7887c6-6fpnn\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.255382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.255344    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:35.321558 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.321518    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw" event={"ID":"cd020c42-b2df-4082-98f5-799fca16f397","Type":"ContainerStarted","Data":"89b4c4ca3a29addeecfd8c4f4183c79db3cef7996edc16ee83ce630829cf413f"}
Nov 03 23:43:35.321710 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.321593    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw" event={"ID":"cd020c42-b2df-4082-98f5-799fca16f397","Type":"ContainerStarted","Data":"5b163975b7d51c81d2fb5e3f14dc73de05fec4a26a6e4420758162c6bdac3472"}
Nov 03 23:43:35.323551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.323494    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" event={"ID":"c631f8d7-c832-4e70-b291-4e8629e024a5","Type":"ContainerStarted","Data":"57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a"}
Nov 03 23:43:35.323551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.323527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" event={"ID":"c631f8d7-c832-4e70-b291-4e8629e024a5","Type":"ContainerStarted","Data":"0dae702945e4d67df117d141c91d3c75d5e40712971151150843a90493301991"}
Nov 03 23:43:35.354609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.353977    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw" podStartSLOduration=2.35395892 podStartE2EDuration="2.35395892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:35.341822755 +0000 UTC m=+880.441422899" watchObservedRunningTime="2026-11-03 23:43:35.35395892 +0000 UTC m=+880.453559066"
Nov 03 23:43:35.368825 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.368775    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podStartSLOduration=3.368759924 podStartE2EDuration="3.368759924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:35.367885715 +0000 UTC m=+880.467485861" watchObservedRunningTime="2026-11-03 23:43:35.368759924 +0000 UTC m=+880.468360069"
Nov 03 23:43:35.567396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:35.567367    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"]
Nov 03 23:43:35.568171 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:35.568125    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod292fcf1a_0854_403f_b440_9878ad3811cf.slice/crio-5d3099c2cfe744855dfad65fda9e19dda66e0f089cf9da60835944ffae439fb7 WatchSource:0}: Error finding container 5d3099c2cfe744855dfad65fda9e19dda66e0f089cf9da60835944ffae439fb7: Status 404 returned error can't find the container with id 5d3099c2cfe744855dfad65fda9e19dda66e0f089cf9da60835944ffae439fb7
Nov 03 23:43:36.282525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:36.282495    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:36.338272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:36.338231    5074 generic.go:356] "Generic (PLEG): container finished" podID="cd020c42-b2df-4082-98f5-799fca16f397" containerID="89b4c4ca3a29addeecfd8c4f4183c79db3cef7996edc16ee83ce630829cf413f" exitCode=0
Nov 03 23:43:36.338616 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:36.338469    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw" event={"ID":"cd020c42-b2df-4082-98f5-799fca16f397","Type":"ContainerDied","Data":"89b4c4ca3a29addeecfd8c4f4183c79db3cef7996edc16ee83ce630829cf413f"}
Nov 03 23:43:36.341629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:36.341597    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" event={"ID":"292fcf1a-0854-403f-b440-9878ad3811cf","Type":"ContainerStarted","Data":"1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b"}
Nov 03 23:43:36.341729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:36.341633    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" event={"ID":"292fcf1a-0854-403f-b440-9878ad3811cf","Type":"ContainerStarted","Data":"5d3099c2cfe744855dfad65fda9e19dda66e0f089cf9da60835944ffae439fb7"}
Nov 03 23:43:36.399088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:36.399035    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podStartSLOduration=2.399021611 podStartE2EDuration="2.399021611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:36.396442722 +0000 UTC m=+881.496042865" watchObservedRunningTime="2026-11-03 23:43:36.399021611 +0000 UTC m=+881.498621754"
Nov 03 23:43:37.256503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.256466    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:37.528250 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.528186    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:37.573332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.573297    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca\") pod \"cd020c42-b2df-4082-98f5-799fca16f397\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") "
Nov 03 23:43:37.573471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.573446    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr\" (UniqueName: \"kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr\") pod \"cd020c42-b2df-4082-98f5-799fca16f397\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") "
Nov 03 23:43:37.573530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.573477    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca\") pod \"cd020c42-b2df-4082-98f5-799fca16f397\" (UID: \"cd020c42-b2df-4082-98f5-799fca16f397\") "
Nov 03 23:43:37.573699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.573674    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca" pod "cd020c42-b2df-4082-98f5-799fca16f397" (UID: "cd020c42-b2df-4082-98f5-799fca16f397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:37.574087 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.574063    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.575878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.575802    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr" pod "cd020c42-b2df-4082-98f5-799fca16f397" (UID: "cd020c42-b2df-4082-98f5-799fca16f397"). InnerVolumeSpecName "kube-api-access-v2ggr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:37.576021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.575999    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca" pod "cd020c42-b2df-4082-98f5-799fca16f397" (UID: "cd020c42-b2df-4082-98f5-799fca16f397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:37.675049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.675011    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ggr\" (UniqueName: \"kubernetes.io/projected/cd020c42-b2df-4082-98f5-799fca16f397-kube-api-access-v2ggr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.675049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:37.675050    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd020c42-b2df-4082-98f5-799fca16f397-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:38.359439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:38.359413    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"
Nov 03 23:43:38.359589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:38.359452    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw" event={"ID":"cd020c42-b2df-4082-98f5-799fca16f397","Type":"ContainerDied","Data":"5b163975b7d51c81d2fb5e3f14dc73de05fec4a26a6e4420758162c6bdac3472"}
Nov 03 23:43:38.359589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:38.359503    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b163975b7d51c81d2fb5e3f14dc73de05fec4a26a6e4420758162c6bdac3472"
Nov 03 23:43:43.282444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.282404    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:43.295336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.295306    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:43.405821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.405786    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"]
Nov 03 23:43:43.406554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.406537    5074 cpu_manager.go:414] "containerMap: removing container" podUID="cd020c42-b2df-4082-98f5-799fca16f397" containerName="revoke-keys"
Nov 03 23:43:43.406554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.406554    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd020c42-b2df-4082-98f5-799fca16f397" containerName="revoke-keys"
Nov 03 23:43:43.406733 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.406724    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd020c42-b2df-4082-98f5-799fca16f397" containerName="revoke-keys"
Nov 03 23:43:43.410796 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.410775    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.411317 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.411289    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:43:43.417064 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.416044    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:43:43.417064 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.416054    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:43:43.429964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.429935    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"]
Nov 03 23:43:43.550943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.550860    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.551091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.550989    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.551091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.551066    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsfb\" (UniqueName: \"kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.652541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.652508    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsfb\" (UniqueName: \"kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.652696 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.652680    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.652778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.652715    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.653486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.653465    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.655257 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.655232    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.666067 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.666041    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsfb\" (UniqueName: \"kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb\") pod \"maas-api-revoke-sub-e7b60f24-mhqtx\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:43.756906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:43.756873    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:44.064793 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:44.063864    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a97108_5998_4704_bf82_d4c0067cdac8.slice/crio-441b7c5c622bbb2bd97f8d9066d671947df506c803aaacbb9af76e3de87b7dbd WatchSource:0}: Error finding container 441b7c5c622bbb2bd97f8d9066d671947df506c803aaacbb9af76e3de87b7dbd: Status 404 returned error can't find the container with id 441b7c5c622bbb2bd97f8d9066d671947df506c803aaacbb9af76e3de87b7dbd
Nov 03 23:43:44.064793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:44.064330    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"]
Nov 03 23:43:44.405639 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:44.405607    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx" event={"ID":"23a97108-5998-4704-bf82-d4c0067cdac8","Type":"ContainerStarted","Data":"441b7c5c622bbb2bd97f8d9066d671947df506c803aaacbb9af76e3de87b7dbd"}
Nov 03 23:43:45.257167 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:45.257134    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:45.275499 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:45.275412    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:45.425980 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:45.425917    5074 generic.go:356] "Generic (PLEG): container finished" podID="23a97108-5998-4704-bf82-d4c0067cdac8" containerID="c34b125c2286c51dab3df35d7e3f57c387061e1f5a0001f1cb3b50c59a505c9c" exitCode=0
Nov 03 23:43:45.426979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:45.426596    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx" event={"ID":"23a97108-5998-4704-bf82-d4c0067cdac8","Type":"ContainerDied","Data":"c34b125c2286c51dab3df35d7e3f57c387061e1f5a0001f1cb3b50c59a505c9c"}
Nov 03 23:43:45.459968 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:45.459941    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:43:47.182766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.181914    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:47.436826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.433394    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb\" (UniqueName: \"kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb\") pod \"23a97108-5998-4704-bf82-d4c0067cdac8\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") "
Nov 03 23:43:47.436826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.433668    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca\") pod \"23a97108-5998-4704-bf82-d4c0067cdac8\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") "
Nov 03 23:43:47.436826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.433771    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca\") pod \"23a97108-5998-4704-bf82-d4c0067cdac8\" (UID: \"23a97108-5998-4704-bf82-d4c0067cdac8\") "
Nov 03 23:43:47.436826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.434958    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca" pod "23a97108-5998-4704-bf82-d4c0067cdac8" (UID: "23a97108-5998-4704-bf82-d4c0067cdac8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:47.547766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.546105    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:47.547766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.546340    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb" pod "23a97108-5998-4704-bf82-d4c0067cdac8" (UID: "23a97108-5998-4704-bf82-d4c0067cdac8"). InnerVolumeSpecName "kube-api-access-rnsfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:47.589752 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.589688    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx" event={"ID":"23a97108-5998-4704-bf82-d4c0067cdac8","Type":"ContainerDied","Data":"441b7c5c622bbb2bd97f8d9066d671947df506c803aaacbb9af76e3de87b7dbd"}
Nov 03 23:43:47.590081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.590057    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441b7c5c622bbb2bd97f8d9066d671947df506c803aaacbb9af76e3de87b7dbd"
Nov 03 23:43:47.590263 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.590242    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"
Nov 03 23:43:47.623763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.622886    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca" pod "23a97108-5998-4704-bf82-d4c0067cdac8" (UID: "23a97108-5998-4704-bf82-d4c0067cdac8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:47.651762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.650718    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsfb\" (UniqueName: \"kubernetes.io/projected/23a97108-5998-4704-bf82-d4c0067cdac8-kube-api-access-rnsfb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:47.651762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:47.650785    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a97108-5998-4704-bf82-d4c0067cdac8-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:49.188848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.188816    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"]
Nov 03 23:43:49.189339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.189326    5074 cpu_manager.go:414] "containerMap: removing container" podUID="23a97108-5998-4704-bf82-d4c0067cdac8" containerName="revoke-keys"
Nov 03 23:43:49.189384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.189340    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a97108-5998-4704-bf82-d4c0067cdac8" containerName="revoke-keys"
Nov 03 23:43:49.189457 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.189448    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a97108-5998-4704-bf82-d4c0067cdac8" containerName="revoke-keys"
Nov 03 23:43:49.192711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.192689    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.214469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.214005    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"]
Nov 03 23:43:49.288027 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.287992    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288027 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288030    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288056    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288161    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288194    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288228    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288376 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288285    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288376 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288337    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.288453 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.288398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389178    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389240    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389270    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389307    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389374    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389408    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389459    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389494    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389539    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389851 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389683    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389851 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389791    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389960 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389856    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.389960 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.389896    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.390080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.390062    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.391846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.391829    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.391992 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.391976    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.407246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.407218    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.407603 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.407584    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.506491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.506400    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:49.800029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.799997    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"]
Nov 03 23:43:49.800706 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:43:49.800678    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8486895c_4686_4f0e_9286_f74f6303ab83.slice/crio-a10c294292e86d2caf15e2f5637d915082872ede5333491383c0c43264c3f788 WatchSource:0}: Error finding container a10c294292e86d2caf15e2f5637d915082872ede5333491383c0c43264c3f788: Status 404 returned error can't find the container with id a10c294292e86d2caf15e2f5637d915082872ede5333491383c0c43264c3f788
Nov 03 23:43:49.803185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.803152    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:49.803290 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.803214    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:49.803290 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:49.803240    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:43:50.634111 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:50.634074    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" event={"ID":"8486895c-4686-4f0e-9286-f74f6303ab83","Type":"ContainerStarted","Data":"54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78"}
Nov 03 23:43:50.634849 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:50.634822    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" event={"ID":"8486895c-4686-4f0e-9286-f74f6303ab83","Type":"ContainerStarted","Data":"a10c294292e86d2caf15e2f5637d915082872ede5333491383c0c43264c3f788"}
Nov 03 23:43:50.686690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:50.686640    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podStartSLOduration=1.6866270700000001 podStartE2EDuration="1.68662707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:50.686016447 +0000 UTC m=+895.785616591" watchObservedRunningTime="2026-11-03 23:43:50.68662707 +0000 UTC m=+895.786227214"
Nov 03 23:43:51.513764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:51.512092    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:51.521762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:51.514638    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 23:43:51.521762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:51.514755    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 23:43:52.508093 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:52.508049    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 23:43:52.508704 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:52.508122    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 23:43:53.520764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:53.520110    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 23:43:53.520764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:53.520186    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 23:43:54.516763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:54.511127    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 23:43:54.516763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:54.511191    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 23:43:55.181757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:55.181704    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:43:55.225765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:55.191308    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:43:55.516763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:55.513090    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 23:43:55.516763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:55.513154    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 23:43:56.171206 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.171158    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"]
Nov 03 23:43:56.177440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.177399    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.184132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.184098    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:43:56.184585 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.184286    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:43:56.185360 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.185323    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"]
Nov 03 23:43:56.194204 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.194163    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggsk\" (UniqueName: \"kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.194399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.194261    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.194475 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.194399    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.296389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.296329    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggsk\" (UniqueName: \"kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.296633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.296425    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.296633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.296533    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.297428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.297395    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.299328 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.299298    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.315359 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.315320    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggsk\" (UniqueName: \"kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk\") pod \"maas-api-revoke-sub-cacd2ccf-zkpbx\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.502578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.502539    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:56.507287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.507254    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 23:43:56.507450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.507317    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 23:43:56.818716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:56.818620    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"]
Nov 03 23:43:57.511928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:57.511888    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:57.512416 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:57.512372    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:57.513106 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:57.513086    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 03 23:43:57.694671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:57.694629    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx" event={"ID":"a45de119-37f8-417f-8e39-603b4c7bdfeb","Type":"ContainerStarted","Data":"1bdfb410683574db3e93223a919d7be30d5fc491037aa9e0d123d62a83e07bb3"}
Nov 03 23:43:57.694858 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:57.694692    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx" event={"ID":"a45de119-37f8-417f-8e39-603b4c7bdfeb","Type":"ContainerStarted","Data":"9b6face8da2f3b04c6e1b249af75bc5747d4bd7ebb2917e5d2170d8b541e4b2c"}
Nov 03 23:43:57.740300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:57.740223    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx" podStartSLOduration=1.740205545 podStartE2EDuration="1.740205545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:57.737407334 +0000 UTC m=+902.837007478" watchObservedRunningTime="2026-11-03 23:43:57.740205545 +0000 UTC m=+902.839805736"
Nov 03 23:43:58.044113 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:43:58.044068    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:43:58.704576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:58.704535    5074 generic.go:356] "Generic (PLEG): container finished" podID="a45de119-37f8-417f-8e39-603b4c7bdfeb" containerID="1bdfb410683574db3e93223a919d7be30d5fc491037aa9e0d123d62a83e07bb3" exitCode=0
Nov 03 23:43:58.705040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:58.704712    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx" event={"ID":"a45de119-37f8-417f-8e39-603b4c7bdfeb","Type":"ContainerDied","Data":"1bdfb410683574db3e93223a919d7be30d5fc491037aa9e0d123d62a83e07bb3"}
Nov 03 23:43:59.882731 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.882703    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:43:59.958884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.958847    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk\" (UniqueName: \"kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk\") pod \"a45de119-37f8-417f-8e39-603b4c7bdfeb\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") "
Nov 03 23:43:59.959086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.959017    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca\") pod \"a45de119-37f8-417f-8e39-603b4c7bdfeb\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") "
Nov 03 23:43:59.959086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.959069    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca\") pod \"a45de119-37f8-417f-8e39-603b4c7bdfeb\" (UID: \"a45de119-37f8-417f-8e39-603b4c7bdfeb\") "
Nov 03 23:43:59.959543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.959513    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca" pod "a45de119-37f8-417f-8e39-603b4c7bdfeb" (UID: "a45de119-37f8-417f-8e39-603b4c7bdfeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:43:59.963643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.963589    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca" pod "a45de119-37f8-417f-8e39-603b4c7bdfeb" (UID: "a45de119-37f8-417f-8e39-603b4c7bdfeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:59.963818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:43:59.963702    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk" pod "a45de119-37f8-417f-8e39-603b4c7bdfeb" (UID: "a45de119-37f8-417f-8e39-603b4c7bdfeb"). InnerVolumeSpecName "kube-api-access-tggsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:00.060697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.060647    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tggsk\" (UniqueName: \"kubernetes.io/projected/a45de119-37f8-417f-8e39-603b4c7bdfeb-kube-api-access-tggsk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:00.060697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.060677    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:00.060697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.060694    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a45de119-37f8-417f-8e39-603b4c7bdfeb-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:00.504133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.504096    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"]
Nov 03 23:44:00.504936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.504908    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a45de119-37f8-417f-8e39-603b4c7bdfeb" containerName="revoke-keys"
Nov 03 23:44:00.504936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.504934    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45de119-37f8-417f-8e39-603b4c7bdfeb" containerName="revoke-keys"
Nov 03 23:44:00.505138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.505123    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45de119-37f8-417f-8e39-603b4c7bdfeb" containerName="revoke-keys"
Nov 03 23:44:00.509331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.509306    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.519589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.519555    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"]
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.568656    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.568874    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.568939    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.569055    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.569169    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.569201    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.569223    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.569262    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w58b\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.569648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.569306    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.675830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.675788    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.676456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.676415    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.676727 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.676711    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.676990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.676963    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.677537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.677510    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.677642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.677567    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.677642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.677598    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.677775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.677642    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w58b\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.677775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.677676    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.677775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.677715    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.685273 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.685249    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.686557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.686487    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.687145 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.687103    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.688425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.688405    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.688836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.688809    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.690845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.690823    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.694311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.694285    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.698357 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.698329    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w58b\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:00.722683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.722642    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx" event={"ID":"a45de119-37f8-417f-8e39-603b4c7bdfeb","Type":"ContainerDied","Data":"9b6face8da2f3b04c6e1b249af75bc5747d4bd7ebb2917e5d2170d8b541e4b2c"}
Nov 03 23:44:00.722683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.722671    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"
Nov 03 23:44:00.722955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.722705    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6face8da2f3b04c6e1b249af75bc5747d4bd7ebb2917e5d2170d8b541e4b2c"
Nov 03 23:44:00.845365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:00.845259    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:01.141279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.141247    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"]
Nov 03 23:44:01.141796 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:01.141768    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3120e42_afef_4b73_812f_85c2614e398c.slice/crio-580a4ab3a7738d8fb62bb2ef22eaa61d42c38b2908d89a8556fc61e44f8567fd WatchSource:0}: Error finding container 580a4ab3a7738d8fb62bb2ef22eaa61d42c38b2908d89a8556fc61e44f8567fd: Status 404 returned error can't find the container with id 580a4ab3a7738d8fb62bb2ef22eaa61d42c38b2908d89a8556fc61e44f8567fd
Nov 03 23:44:01.144894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.144863    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:01.145005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.144921    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:01.145005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.144949    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:01.760762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.759458    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" event={"ID":"c3120e42-afef-4b73-812f-85c2614e398c","Type":"ContainerStarted","Data":"1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d"}
Nov 03 23:44:01.760762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.759545    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" event={"ID":"c3120e42-afef-4b73-812f-85c2614e398c","Type":"ContainerStarted","Data":"580a4ab3a7738d8fb62bb2ef22eaa61d42c38b2908d89a8556fc61e44f8567fd"}
Nov 03 23:44:01.808761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.805636    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podStartSLOduration=1.805613632 podStartE2EDuration="1.805613632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:01.795004274 +0000 UTC m=+906.894604430" watchObservedRunningTime="2026-11-03 23:44:01.805613632 +0000 UTC m=+906.905213779"
Nov 03 23:44:01.845377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:01.845339    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:02.572589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.572540    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"]
Nov 03 23:44:02.754146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.754114    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"]
Nov 03 23:44:02.760941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.754433    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:02.927814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.835176    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:02.928169 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.928135    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:02.966895 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.852792    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 23:44:02.967187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.967146    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 23:44:02.968152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.968127    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:02.968820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.968798    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:02.968986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.968967    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrj7b\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.009810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:02.977905    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.010102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.010079    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.010329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.010310    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.010488 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.010471    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.122764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122468    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.122764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122577    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.122764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122654    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.122764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122706    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.122764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122728    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrj7b\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.123205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122798    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.123205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122842    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.123205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122939    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.123205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.122986    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.126764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.124094    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.127974 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.127948    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.140848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.128802    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.141035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.136587    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.141143 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.130726    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.198814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.188344    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.198814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.191032    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.207947 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.207915    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrj7b\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.225835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.225786    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.335977 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.335943    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:03.869563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.869527    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 23:44:03.872259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:03.872227    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 23:44:04.380066 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:04.380013    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"]
Nov 03 23:44:04.393131 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:04.393060    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16f47b3_31ad_4e27_bdc8_aa26c334ab54.slice/crio-fc4d5a337b708189173876dc9a2e1ba71abc9c08f42d4214912be49d390a86df WatchSource:0}: Error finding container fc4d5a337b708189173876dc9a2e1ba71abc9c08f42d4214912be49d390a86df: Status 404 returned error can't find the container with id fc4d5a337b708189173876dc9a2e1ba71abc9c08f42d4214912be49d390a86df
Nov 03 23:44:04.395192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:04.395147    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:04.395319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:04.395223    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:04.395319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:04.395264    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:04.847530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:04.847089    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 23:44:04.847530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:04.847150    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 23:44:05.113825 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.112160    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" event={"ID":"f16f47b3-31ad-4e27-bdc8-aa26c334ab54","Type":"ContainerStarted","Data":"889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a"}
Nov 03 23:44:05.113825 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.112230    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" event={"ID":"f16f47b3-31ad-4e27-bdc8-aa26c334ab54","Type":"ContainerStarted","Data":"fc4d5a337b708189173876dc9a2e1ba71abc9c08f42d4214912be49d390a86df"}
Nov 03 23:44:05.149233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.149162    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podStartSLOduration=3.149144845 podStartE2EDuration="3.149144845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:05.141623505 +0000 UTC m=+910.241223663" watchObservedRunningTime="2026-11-03 23:44:05.149144845 +0000 UTC m=+910.248744992"
Nov 03 23:44:05.336930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.336893    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:05.338621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.338577    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:05.338784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.338650    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:05.866760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.856179    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 23:44:05.866760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:05.856242    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 23:44:06.344772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:06.344157    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:06.344772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:06.344234    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:06.846775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:06.846184    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 23:44:06.846775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:06.846250    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 23:44:07.072147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.055801    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"]
Nov 03 23:44:07.077717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.077686    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.078434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.078379    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"]
Nov 03 23:44:07.085014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.084990    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:44:07.085138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.085114    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:44:07.091952 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.091914    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.092135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.092104    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn259\" (UniqueName: \"kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.092398 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.092369    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.193132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.193094    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.193132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.193134    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn259\" (UniqueName: \"kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.193324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.193220    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.194189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.194167    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.195666 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.195647    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.282763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.281337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn259\" (UniqueName: \"kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259\") pod \"maas-api-revoke-sub-2f8e5d23-ktsnp\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.337765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.337163    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:07.337765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.337230    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:07.397345 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.397303    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:07.843493 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.842602    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"]
Nov 03 23:44:07.846008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.845775    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 23:44:07.846008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.845816    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 23:44:07.852169 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.852139    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"]
Nov 03 23:44:07.877359 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.876933    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.877915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.876834    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"]
Nov 03 23:44:07.939626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.939584    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhdp\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.942091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.942053    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.942328 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.942310    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.942519 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.942490    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.942752 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.942719    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.943074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.943039    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.943236 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.943219    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.943461 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.943444    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:07.943683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:07.943666    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.044830    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhdp\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.044882    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.044924    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.044972    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.044997    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.045098    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.045127    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.045209    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.045761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.045270    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.046391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.045981    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.046391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.046040    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.046391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.046238    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.048406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.046517    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.048406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.047039    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.049607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.049288    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.049836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.049727    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.062411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.062384    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.062521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.062390    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhdp\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.190316 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.190278    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp" event={"ID":"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06","Type":"ContainerStarted","Data":"a9867ee8e1c541fca41237c543d739d51fb6320c70002dab0bbcd1bcc6d29c49"}
Nov 03 23:44:08.246203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.246164    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:08.339101 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.339063    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:08.339277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.339127    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:08.583037 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.583011    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"]
Nov 03 23:44:08.586232 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:08.586204    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa649ec_dbbe_4b98_8287_da4b1f7e7fdc.slice/crio-b3cdc55ae5a6797bd2a9fa0324ac7f97c9859d79b7d1c85ebe12a314022f635a WatchSource:0}: Error finding container b3cdc55ae5a6797bd2a9fa0324ac7f97c9859d79b7d1c85ebe12a314022f635a: Status 404 returned error can't find the container with id b3cdc55ae5a6797bd2a9fa0324ac7f97c9859d79b7d1c85ebe12a314022f635a
Nov 03 23:44:08.588938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.588907    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:08.589024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.588974    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:08.589024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:08.589001    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:09.203761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.200784    5074 generic.go:356] "Generic (PLEG): container finished" podID="ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" containerID="65d35adfab4193f54f8071b2dcdd7e62c25d5b5dced5d6a286578421a2760fd5" exitCode=0
Nov 03 23:44:09.203761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.200976    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp" event={"ID":"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06","Type":"ContainerDied","Data":"65d35adfab4193f54f8071b2dcdd7e62c25d5b5dced5d6a286578421a2760fd5"}
Nov 03 23:44:09.203761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.203247    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" event={"ID":"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc","Type":"ContainerStarted","Data":"18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d"}
Nov 03 23:44:09.203761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.203295    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" event={"ID":"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc","Type":"ContainerStarted","Data":"b3cdc55ae5a6797bd2a9fa0324ac7f97c9859d79b7d1c85ebe12a314022f635a"}
Nov 03 23:44:09.261335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.252849    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:09.261335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.259096    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:09.261335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.259146    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:09.341351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.340920    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:09.341351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.340981    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:09.846884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.846844    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:09.847487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:09.847457    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:10.250573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:10.250141    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:10.250573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:10.250200    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:10.342852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:10.341164    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:10.342852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:10.341225    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:10.909860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:10.909824    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:10.910409 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:10.910375    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:11.044029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.043499    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:11.105845 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.101162    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podStartSLOduration=4.101141181 podStartE2EDuration="4.101141181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:09.28540582 +0000 UTC m=+914.385005965" watchObservedRunningTime="2026-11-03 23:44:11.101141181 +0000 UTC m=+916.200741325"
Nov 03 23:44:11.153612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.153570    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca\") pod \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") "
Nov 03 23:44:11.160269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.160237    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259\" (UniqueName: \"kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259\") pod \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") "
Nov 03 23:44:11.163939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.160573    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca\") pod \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\" (UID: \"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06\") "
Nov 03 23:44:11.193770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.167933    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca" pod "ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" (UID: "ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:11.217955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.212977    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca" pod "ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" (UID: "ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:11.217955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.213467    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259" pod "ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" (UID: "ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06"). InnerVolumeSpecName "kube-api-access-fn259". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:11.288764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.265194    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:11.309022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.308979    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:11.337199 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.312995    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:11.337423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.337405    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn259\" (UniqueName: \"kubernetes.io/projected/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-kube-api-access-fn259\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:11.337538 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.337526    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:11.346348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.346311    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:44:11.346611 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.346580    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 23:44:11.454813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.452458    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"
Nov 03 23:44:11.468181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.468145    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp" event={"ID":"ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06","Type":"ContainerDied","Data":"a9867ee8e1c541fca41237c543d739d51fb6320c70002dab0bbcd1bcc6d29c49"}
Nov 03 23:44:11.468405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.468386    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9867ee8e1c541fca41237c543d739d51fb6320c70002dab0bbcd1bcc6d29c49"
Nov 03 23:44:11.762725 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.762641    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:11.763597 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.763575    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:11.764483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:11.764461    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 03 23:44:12.249959 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:12.247819    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:12.249959 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:12.247881    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:12.557227 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:12.557132    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:12.557614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:12.557583    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:12.558463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:12.558440    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 03 23:44:13.247707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:13.247661    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:13.247942 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:13.247732    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:14.247369 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:14.247331    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:14.247919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:14.247392    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:15.247769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:15.247715    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 23:44:15.248267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:15.247793    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 23:44:16.532564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:16.532524    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:16.533447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:16.533426    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:16.534554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:16.534507    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 03 23:44:17.353853 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.353817    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"]
Nov 03 23:44:17.354411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.354398    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" containerName="revoke-keys"
Nov 03 23:44:17.354455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.354413    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" containerName="revoke-keys"
Nov 03 23:44:17.354533 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.354525    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" containerName="revoke-keys"
Nov 03 23:44:17.357595 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.357578    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.367197 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.367175    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:44:17.367477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.367462    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:44:17.376365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.376340    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"]
Nov 03 23:44:17.470101 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.470069    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.470264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.470195    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.470264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.470229    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqfx\" (UniqueName: \"kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.571656 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.571629    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.572155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.571727    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.572155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.571774    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqfx\" (UniqueName: \"kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.572526 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.572499    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.574259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.574238    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.592694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.592656    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqfx\" (UniqueName: \"kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx\") pod \"maas-api-revoke-sub-f1400539-9k96l\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.670387 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.670309    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:17.959372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:17.959343    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"]
Nov 03 23:44:17.959718 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:17.959696    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f040f79_9279_4e90_8b23_833ddabc12d0.slice/crio-ac315962a7b71f9e80a9ef2d8d510c92041ae5b1668e041e6f026d2c4cae090d WatchSource:0}: Error finding container ac315962a7b71f9e80a9ef2d8d510c92041ae5b1668e041e6f026d2c4cae090d: Status 404 returned error can't find the container with id ac315962a7b71f9e80a9ef2d8d510c92041ae5b1668e041e6f026d2c4cae090d
Nov 03 23:44:18.506031 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:18.505925    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l" event={"ID":"4f040f79-9279-4e90-8b23-833ddabc12d0","Type":"ContainerStarted","Data":"0f9d6b155463477d6e63c5a5e8af9da3d0b524ed02487d468e3569b1f7a3f72b"}
Nov 03 23:44:18.506879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:18.506848    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l" event={"ID":"4f040f79-9279-4e90-8b23-833ddabc12d0","Type":"ContainerStarted","Data":"ac315962a7b71f9e80a9ef2d8d510c92041ae5b1668e041e6f026d2c4cae090d"}
Nov 03 23:44:18.537921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:18.537857    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l" podStartSLOduration=1.537843071 podStartE2EDuration="1.537843071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:18.534284207 +0000 UTC m=+923.633884353" watchObservedRunningTime="2026-11-03 23:44:18.537843071 +0000 UTC m=+923.637443218"
Nov 03 23:44:19.516773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.516700    5074 generic.go:356] "Generic (PLEG): container finished" podID="4f040f79-9279-4e90-8b23-833ddabc12d0" containerID="0f9d6b155463477d6e63c5a5e8af9da3d0b524ed02487d468e3569b1f7a3f72b" exitCode=0
Nov 03 23:44:19.517270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.516789    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l" event={"ID":"4f040f79-9279-4e90-8b23-833ddabc12d0","Type":"ContainerDied","Data":"0f9d6b155463477d6e63c5a5e8af9da3d0b524ed02487d468e3569b1f7a3f72b"}
Nov 03 23:44:19.566542 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.566509    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"]
Nov 03 23:44:19.569672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.569647    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.580905 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.580879    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"]
Nov 03 23:44:19.697540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.697513    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.697699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.697596    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.697699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.697691    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmzl\" (UniqueName: \"kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.799614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.799519    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.799785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.799653    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmzl\" (UniqueName: \"kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.799785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.799777    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.800331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.800306    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.802202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.802180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.813389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.813363    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmzl\" (UniqueName: \"kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl\") pod \"maas-api-revoke-sub-630011a8-9nxzn\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:19.886644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:19.886611    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:20.173645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.173614    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"]
Nov 03 23:44:20.174082 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:20.174057    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda764d63b_1c3f_4578_9bbe_9b06cb89ba03.slice/crio-0bcc973445d6e3898a17e15a26d59185fc607b49d519ba61ecb20c2c97ad688e WatchSource:0}: Error finding container 0bcc973445d6e3898a17e15a26d59185fc607b49d519ba61ecb20c2c97ad688e: Status 404 returned error can't find the container with id 0bcc973445d6e3898a17e15a26d59185fc607b49d519ba61ecb20c2c97ad688e
Nov 03 23:44:20.526211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.526174    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn" event={"ID":"a764d63b-1c3f-4578-9bbe-9b06cb89ba03","Type":"ContainerStarted","Data":"b64fe1cb83d352e7bf791a4e45f23a2e70297667ba5c4862edd2fe0814cddcaa"}
Nov 03 23:44:20.526608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.526241    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn" event={"ID":"a764d63b-1c3f-4578-9bbe-9b06cb89ba03","Type":"ContainerStarted","Data":"0bcc973445d6e3898a17e15a26d59185fc607b49d519ba61ecb20c2c97ad688e"}
Nov 03 23:44:20.564360 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.564305    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn" podStartSLOduration=1.564286778 podStartE2EDuration="1.564286778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:20.560515904 +0000 UTC m=+925.660116047" watchObservedRunningTime="2026-11-03 23:44:20.564286778 +0000 UTC m=+925.663886921"
Nov 03 23:44:20.663937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.663910    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:20.713937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.713891    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca\") pod \"4f040f79-9279-4e90-8b23-833ddabc12d0\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") "
Nov 03 23:44:20.715506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.714529    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx\" (UniqueName: \"kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx\") pod \"4f040f79-9279-4e90-8b23-833ddabc12d0\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") "
Nov 03 23:44:20.715506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.714596    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca\") pod \"4f040f79-9279-4e90-8b23-833ddabc12d0\" (UID: \"4f040f79-9279-4e90-8b23-833ddabc12d0\") "
Nov 03 23:44:20.717428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.717397    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca" pod "4f040f79-9279-4e90-8b23-833ddabc12d0" (UID: "4f040f79-9279-4e90-8b23-833ddabc12d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:20.717942 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.717908    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx" pod "4f040f79-9279-4e90-8b23-833ddabc12d0" (UID: "4f040f79-9279-4e90-8b23-833ddabc12d0"). InnerVolumeSpecName "kube-api-access-ddqfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:20.719067 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.719040    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca" pod "4f040f79-9279-4e90-8b23-833ddabc12d0" (UID: "4f040f79-9279-4e90-8b23-833ddabc12d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:20.816344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.816315    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:20.816344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.816343    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddqfx\" (UniqueName: \"kubernetes.io/projected/4f040f79-9279-4e90-8b23-833ddabc12d0-kube-api-access-ddqfx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:20.816496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:20.816354    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f040f79-9279-4e90-8b23-833ddabc12d0-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:21.545759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:21.544766    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l" event={"ID":"4f040f79-9279-4e90-8b23-833ddabc12d0","Type":"ContainerDied","Data":"ac315962a7b71f9e80a9ef2d8d510c92041ae5b1668e041e6f026d2c4cae090d"}
Nov 03 23:44:21.545759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:21.544832    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac315962a7b71f9e80a9ef2d8d510c92041ae5b1668e041e6f026d2c4cae090d"
Nov 03 23:44:21.545759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:21.544906    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"
Nov 03 23:44:21.556425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:21.552767    5074 generic.go:356] "Generic (PLEG): container finished" podID="a764d63b-1c3f-4578-9bbe-9b06cb89ba03" containerID="b64fe1cb83d352e7bf791a4e45f23a2e70297667ba5c4862edd2fe0814cddcaa" exitCode=0
Nov 03 23:44:21.556425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:21.552958    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn" event={"ID":"a764d63b-1c3f-4578-9bbe-9b06cb89ba03","Type":"ContainerDied","Data":"b64fe1cb83d352e7bf791a4e45f23a2e70297667ba5c4862edd2fe0814cddcaa"}
Nov 03 23:44:23.656118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.656071    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn" event={"ID":"a764d63b-1c3f-4578-9bbe-9b06cb89ba03","Type":"ContainerDied","Data":"0bcc973445d6e3898a17e15a26d59185fc607b49d519ba61ecb20c2c97ad688e"}
Nov 03 23:44:23.656710 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.656685    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bcc973445d6e3898a17e15a26d59185fc607b49d519ba61ecb20c2c97ad688e"
Nov 03 23:44:23.660970 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.660944    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:23.700640 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.700587    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca\") pod \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") "
Nov 03 23:44:23.702145 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.702118    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca" pod "a764d63b-1c3f-4578-9bbe-9b06cb89ba03" (UID: "a764d63b-1c3f-4578-9bbe-9b06cb89ba03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:23.702313 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.701729    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca\") pod \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") "
Nov 03 23:44:23.702541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.702525    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl\" (UniqueName: \"kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl\") pod \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\" (UID: \"a764d63b-1c3f-4578-9bbe-9b06cb89ba03\") "
Nov 03 23:44:23.703430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.703411    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:23.716630 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.716385    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca" pod "a764d63b-1c3f-4578-9bbe-9b06cb89ba03" (UID: "a764d63b-1c3f-4578-9bbe-9b06cb89ba03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:23.716630 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.716570    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl" pod "a764d63b-1c3f-4578-9bbe-9b06cb89ba03" (UID: "a764d63b-1c3f-4578-9bbe-9b06cb89ba03"). InnerVolumeSpecName "kube-api-access-7fmzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:23.836829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.809064    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:23.836829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:23.809105    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmzl\" (UniqueName: \"kubernetes.io/projected/a764d63b-1c3f-4578-9bbe-9b06cb89ba03-kube-api-access-7fmzl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:24.663363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:24.663335    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"
Nov 03 23:44:26.032189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.032152    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"]
Nov 03 23:44:26.033006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.032984    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a764d63b-1c3f-4578-9bbe-9b06cb89ba03" containerName="revoke-keys"
Nov 03 23:44:26.033071 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.033008    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a764d63b-1c3f-4578-9bbe-9b06cb89ba03" containerName="revoke-keys"
Nov 03 23:44:26.033071 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.033060    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4f040f79-9279-4e90-8b23-833ddabc12d0" containerName="revoke-keys"
Nov 03 23:44:26.033071 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.033068    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f040f79-9279-4e90-8b23-833ddabc12d0" containerName="revoke-keys"
Nov 03 23:44:26.033207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.033195    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a764d63b-1c3f-4578-9bbe-9b06cb89ba03" containerName="revoke-keys"
Nov 03 23:44:26.033242 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.033218    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f040f79-9279-4e90-8b23-833ddabc12d0" containerName="revoke-keys"
Nov 03 23:44:26.037561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.037538    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.045488 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.044041    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:44:26.046323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.046267    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:44:26.059127 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.059102    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"]
Nov 03 23:44:26.144511 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.144474    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.144709 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.144609    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7w2\" (UniqueName: \"kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.144709 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.144695    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.245718 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.245672    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.245938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.245754    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7w2\" (UniqueName: \"kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.245938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.245792    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.246447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.246420    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.248117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.248095    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.265668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.265639    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7w2\" (UniqueName: \"kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2\") pod \"maas-api-revoke-sub-9d0f8676-jxwvt\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.353126 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.353044    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:26.654577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.654538    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"]
Nov 03 23:44:26.680483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:26.680448    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt" event={"ID":"596191db-dd36-4d68-bc86-69878b1bfae4","Type":"ContainerStarted","Data":"59de88a4a90255e867f1fdb6c684ecf7a1fb598508665d0f3243102d43ef37ef"}
Nov 03 23:44:27.690187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:27.690155    5074 generic.go:356] "Generic (PLEG): container finished" podID="596191db-dd36-4d68-bc86-69878b1bfae4" containerID="77d2f5a0a5db1cc8ccf7f26e7db41510cc1139f46e811038df7b3d681a963515" exitCode=0
Nov 03 23:44:27.690598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:27.690249    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt" event={"ID":"596191db-dd36-4d68-bc86-69878b1bfae4","Type":"ContainerDied","Data":"77d2f5a0a5db1cc8ccf7f26e7db41510cc1139f46e811038df7b3d681a963515"}
Nov 03 23:44:28.076088 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:44:28.076049    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:44:28.855841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.855819    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:28.877717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.877690    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2\" (UniqueName: \"kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2\") pod \"596191db-dd36-4d68-bc86-69878b1bfae4\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") "
Nov 03 23:44:28.877883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.877838    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca\") pod \"596191db-dd36-4d68-bc86-69878b1bfae4\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") "
Nov 03 23:44:28.878011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.877980    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca\") pod \"596191db-dd36-4d68-bc86-69878b1bfae4\" (UID: \"596191db-dd36-4d68-bc86-69878b1bfae4\") "
Nov 03 23:44:28.878663 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.878631    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca" pod "596191db-dd36-4d68-bc86-69878b1bfae4" (UID: "596191db-dd36-4d68-bc86-69878b1bfae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:28.880475 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.880431    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2" pod "596191db-dd36-4d68-bc86-69878b1bfae4" (UID: "596191db-dd36-4d68-bc86-69878b1bfae4"). InnerVolumeSpecName "kube-api-access-qd7w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:28.880573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.880521    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca" pod "596191db-dd36-4d68-bc86-69878b1bfae4" (UID: "596191db-dd36-4d68-bc86-69878b1bfae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:28.980149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.980071    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:28.980149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.980104    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596191db-dd36-4d68-bc86-69878b1bfae4-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:28.980149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:28.980115    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd7w2\" (UniqueName: \"kubernetes.io/projected/596191db-dd36-4d68-bc86-69878b1bfae4-kube-api-access-qd7w2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:29.061310 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:44:29.061272    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596191db_dd36_4d68_bc86_69878b1bfae4.slice/crio-59de88a4a90255e867f1fdb6c684ecf7a1fb598508665d0f3243102d43ef37ef\": RecentStats: unable to find data in memory cache]"
Nov 03 23:44:29.276617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.276527    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"]
Nov 03 23:44:29.277222 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.277199    5074 cpu_manager.go:414] "containerMap: removing container" podUID="596191db-dd36-4d68-bc86-69878b1bfae4" containerName="revoke-keys"
Nov 03 23:44:29.277343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.277223    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596191db-dd36-4d68-bc86-69878b1bfae4" containerName="revoke-keys"
Nov 03 23:44:29.277403 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.277391    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="596191db-dd36-4d68-bc86-69878b1bfae4" containerName="revoke-keys"
Nov 03 23:44:29.281338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.281316    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.296677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.296652    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"]
Nov 03 23:44:29.384300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.384267    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9lq\" (UniqueName: \"kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.384499 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.384401    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.384623 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.384598    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.486133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.486099    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9lq\" (UniqueName: \"kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.486345 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.486203    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.486419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.486344    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.487400 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.487370    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.488936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.488909    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.505019 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.504992    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9lq\" (UniqueName: \"kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq\") pod \"maas-api-revoke-sub-e6ebbc6d-8sq7v\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.609663 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.609590    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:29.707801 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.707774    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"
Nov 03 23:44:29.707801 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.707775    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt" event={"ID":"596191db-dd36-4d68-bc86-69878b1bfae4","Type":"ContainerDied","Data":"59de88a4a90255e867f1fdb6c684ecf7a1fb598508665d0f3243102d43ef37ef"}
Nov 03 23:44:29.708044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.707875    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59de88a4a90255e867f1fdb6c684ecf7a1fb598508665d0f3243102d43ef37ef"
Nov 03 23:44:29.911233 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:29.911203    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7969f1_2a24_4327_8aae_162a24fc13d3.slice/crio-d36c5861a305de2a236e82950158793aab7a82581cb6aca8bb62e84a98aebd75 WatchSource:0}: Error finding container d36c5861a305de2a236e82950158793aab7a82581cb6aca8bb62e84a98aebd75: Status 404 returned error can't find the container with id d36c5861a305de2a236e82950158793aab7a82581cb6aca8bb62e84a98aebd75
Nov 03 23:44:29.916348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:29.916301    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"]
Nov 03 23:44:30.721302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:30.721253    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v" event={"ID":"2a7969f1-2a24-4327-8aae-162a24fc13d3","Type":"ContainerStarted","Data":"98a4cef26ede6c1c4d173c60e078b32717e3063cbf162cb6eeba2b81b7e64270"}
Nov 03 23:44:30.721434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:30.721330    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v" event={"ID":"2a7969f1-2a24-4327-8aae-162a24fc13d3","Type":"ContainerStarted","Data":"d36c5861a305de2a236e82950158793aab7a82581cb6aca8bb62e84a98aebd75"}
Nov 03 23:44:30.744010 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:30.743950    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v" podStartSLOduration=1.743938555 podStartE2EDuration="1.743938555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:30.740253656 +0000 UTC m=+935.839853801" watchObservedRunningTime="2026-11-03 23:44:30.743938555 +0000 UTC m=+935.843538711"
Nov 03 23:44:31.731155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:31.731121    5074 generic.go:356] "Generic (PLEG): container finished" podID="2a7969f1-2a24-4327-8aae-162a24fc13d3" containerID="98a4cef26ede6c1c4d173c60e078b32717e3063cbf162cb6eeba2b81b7e64270" exitCode=0
Nov 03 23:44:31.731565 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:31.731192    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v" event={"ID":"2a7969f1-2a24-4327-8aae-162a24fc13d3","Type":"ContainerDied","Data":"98a4cef26ede6c1c4d173c60e078b32717e3063cbf162cb6eeba2b81b7e64270"}
Nov 03 23:44:32.901776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.901727    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:32.932237 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.932204    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq\" (UniqueName: \"kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq\") pod \"2a7969f1-2a24-4327-8aae-162a24fc13d3\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") "
Nov 03 23:44:32.932382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.932272    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca\") pod \"2a7969f1-2a24-4327-8aae-162a24fc13d3\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") "
Nov 03 23:44:32.932471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.932455    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca\") pod \"2a7969f1-2a24-4327-8aae-162a24fc13d3\" (UID: \"2a7969f1-2a24-4327-8aae-162a24fc13d3\") "
Nov 03 23:44:32.933136 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.932719    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca" pod "2a7969f1-2a24-4327-8aae-162a24fc13d3" (UID: "2a7969f1-2a24-4327-8aae-162a24fc13d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:32.933257 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.933169    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:32.935317 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.935289    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca" pod "2a7969f1-2a24-4327-8aae-162a24fc13d3" (UID: "2a7969f1-2a24-4327-8aae-162a24fc13d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:32.936722 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:32.936425    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq" pod "2a7969f1-2a24-4327-8aae-162a24fc13d3" (UID: "2a7969f1-2a24-4327-8aae-162a24fc13d3"). InnerVolumeSpecName "kube-api-access-wk9lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:33.034578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:33.034547    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7969f1-2a24-4327-8aae-162a24fc13d3-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:33.034578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:33.034579    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9lq\" (UniqueName: \"kubernetes.io/projected/2a7969f1-2a24-4327-8aae-162a24fc13d3-kube-api-access-wk9lq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:33.749326 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:33.749291    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"
Nov 03 23:44:33.749500 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:33.749320    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v" event={"ID":"2a7969f1-2a24-4327-8aae-162a24fc13d3","Type":"ContainerDied","Data":"d36c5861a305de2a236e82950158793aab7a82581cb6aca8bb62e84a98aebd75"}
Nov 03 23:44:33.749500 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:33.749367    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d36c5861a305de2a236e82950158793aab7a82581cb6aca8bb62e84a98aebd75"
Nov 03 23:44:39.666626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.666535    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"]
Nov 03 23:44:39.667423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.667405    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2a7969f1-2a24-4327-8aae-162a24fc13d3" containerName="revoke-keys"
Nov 03 23:44:39.667618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.667425    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7969f1-2a24-4327-8aae-162a24fc13d3" containerName="revoke-keys"
Nov 03 23:44:39.667618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.667563    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7969f1-2a24-4327-8aae-162a24fc13d3" containerName="revoke-keys"
Nov 03 23:44:39.671968 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.671946    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.678023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.677999    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:44:39.678945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.678913    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"]
Nov 03 23:44:39.684514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.683667    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:44:39.722233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.721694    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddtz\" (UniqueName: \"kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.722233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.721857    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.722233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.722051    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.834562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.834525    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddtz\" (UniqueName: \"kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.834813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.834711    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.834956 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.834934    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.835951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.835927    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.845394 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.845038    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.848969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.848421    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddtz\" (UniqueName: \"kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz\") pod \"maas-api-revoke-sub-5f3d0b92-bvg4d\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:39.994975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:39.994936    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:40.843764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:40.842785    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"]
Nov 03 23:44:40.912812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:40.912720    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d" event={"ID":"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea","Type":"ContainerStarted","Data":"8ac7da973cf7c7344d52c5a1ce2c34fb07e766335da38a1344aa46d3f9dc903a"}
Nov 03 23:44:41.979763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:41.978694    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d" event={"ID":"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea","Type":"ContainerStarted","Data":"9a7a54e121ad15bf22d6281d695a7a7e3055cd04722e72c1da97cfc4f2376a8c"}
Nov 03 23:44:42.004765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:42.003813    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.113979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:42.113935    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.729557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:42.684269    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.854566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:42.854201    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:42.854566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:42.854268    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.013819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:42.975846    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.137764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.126960    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.183763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.173266    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.330814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.313056    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:43.330814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.313148    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.500861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.500820    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:44:43.501122 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.501108    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:44:43.540192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.526048    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Nov 03 23:44:43.556932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.546268    5074 generic.go:356] "Generic (PLEG): container finished" podID="aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" containerID="9a7a54e121ad15bf22d6281d695a7a7e3055cd04722e72c1da97cfc4f2376a8c" exitCode=0
Nov 03 23:44:43.556932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.546514    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d" event={"ID":"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea","Type":"ContainerDied","Data":"9a7a54e121ad15bf22d6281d695a7a7e3055cd04722e72c1da97cfc4f2376a8c"}
Nov 03 23:44:43.577671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.577641    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:44:43.617962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.616357    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:43.618241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.618215    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.618625 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.617781    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d" podStartSLOduration=4.617761133 podStartE2EDuration="4.617761133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:42.061077809 +0000 UTC m=+947.160677955" watchObservedRunningTime="2026-11-03 23:44:43.617761133 +0000 UTC m=+948.717361277"
Nov 03 23:44:43.685423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:43.664958    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:44:44.373702 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.373661    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"]
Nov 03 23:44:44.378355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.378337    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.386114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.386089    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"]
Nov 03 23:44:44.480455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.480422    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.480617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.480563    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.480666 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.480650    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfn6z\" (UniqueName: \"kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.582202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.582162    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfn6z\" (UniqueName: \"kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.582368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.582310    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.582472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.582433    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.583198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.583168    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.591909 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.585006    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.596753 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.596717    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfn6z\" (UniqueName: \"kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z\") pod \"maas-api-revoke-sub-0630113b-4ws5t\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.695584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.695553    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:44.783408 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.783370    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:44.885818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.885781    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca\") pod \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") "
Nov 03 23:44:44.885998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.885866    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz\" (UniqueName: \"kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz\") pod \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") "
Nov 03 23:44:44.885998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.885951    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca\") pod \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\" (UID: \"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea\") "
Nov 03 23:44:44.886169 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.886141    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca" pod "aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" (UID: "aef2abca-9f4b-42e6-ae8e-13bf3c3849ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:44.886430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.886414    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:44.888010 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.887981    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca" pod "aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" (UID: "aef2abca-9f4b-42e6-ae8e-13bf3c3849ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:44.888123 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.888090    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz" pod "aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" (UID: "aef2abca-9f4b-42e6-ae8e-13bf3c3849ea"). InnerVolumeSpecName "kube-api-access-mddtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:44.987848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.987768    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mddtz\" (UniqueName: \"kubernetes.io/projected/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-kube-api-access-mddtz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:44.987848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:44.987796    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:45.029125 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.029089    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"]
Nov 03 23:44:45.563207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.563172    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d" event={"ID":"aef2abca-9f4b-42e6-ae8e-13bf3c3849ea","Type":"ContainerDied","Data":"8ac7da973cf7c7344d52c5a1ce2c34fb07e766335da38a1344aa46d3f9dc903a"}
Nov 03 23:44:45.563691 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.563230    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac7da973cf7c7344d52c5a1ce2c34fb07e766335da38a1344aa46d3f9dc903a"
Nov 03 23:44:45.563691 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.563181    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"
Nov 03 23:44:45.565030 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.565005    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t" event={"ID":"417e1195-64de-4597-96d4-b8ef49c2bc95","Type":"ContainerStarted","Data":"619d2752911468e0d2aa2bae748503e5c0ebb9b2d53885b7c68eac356310fa31"}
Nov 03 23:44:45.565165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.565043    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t" event={"ID":"417e1195-64de-4597-96d4-b8ef49c2bc95","Type":"ContainerStarted","Data":"857cf5474c318ba45c3469fa17723bc4f9fdde1456eff53b3bb6abac0734b44e"}
Nov 03 23:44:45.591166 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:45.591114    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t" podStartSLOduration=1.59109926 podStartE2EDuration="1.59109926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:45.584935323 +0000 UTC m=+950.684535460" watchObservedRunningTime="2026-11-03 23:44:45.59109926 +0000 UTC m=+950.690699405"
Nov 03 23:44:46.577394 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:46.577317    5074 generic.go:356] "Generic (PLEG): container finished" podID="417e1195-64de-4597-96d4-b8ef49c2bc95" containerID="619d2752911468e0d2aa2bae748503e5c0ebb9b2d53885b7c68eac356310fa31" exitCode=0
Nov 03 23:44:46.577839 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:46.577428    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t" event={"ID":"417e1195-64de-4597-96d4-b8ef49c2bc95","Type":"ContainerDied","Data":"619d2752911468e0d2aa2bae748503e5c0ebb9b2d53885b7c68eac356310fa31"}
Nov 03 23:44:47.614226 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.614181    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"]
Nov 03 23:44:47.615181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.615156    5074 cpu_manager.go:414] "containerMap: removing container" podUID="aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" containerName="revoke-keys"
Nov 03 23:44:47.615277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.615183    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" containerName="revoke-keys"
Nov 03 23:44:47.615383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.615369    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" containerName="revoke-keys"
Nov 03 23:44:47.620563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.619789    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.641346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.640929    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"]
Nov 03 23:44:47.726356 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.726319    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.726645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.726391    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.726645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.726612    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xw8\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.726779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.726686    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.727116 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.727090    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.727177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.727131    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.727213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.727177    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.727213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.727197    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.727273 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.727233    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.780598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.780568    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:47.828336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828305    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca\") pod \"417e1195-64de-4597-96d4-b8ef49c2bc95\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") "
Nov 03 23:44:47.828506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828342    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z\" (UniqueName: \"kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z\") pod \"417e1195-64de-4597-96d4-b8ef49c2bc95\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") "
Nov 03 23:44:47.828506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828388    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca\") pod \"417e1195-64de-4597-96d4-b8ef49c2bc95\" (UID: \"417e1195-64de-4597-96d4-b8ef49c2bc95\") "
Nov 03 23:44:47.828653 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828619    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.828734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828705    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xw8\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.828826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828815    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.828911 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.828895    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829123    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829243    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829303    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca" pod "417e1195-64de-4597-96d4-b8ef49c2bc95" (UID: "417e1195-64de-4597-96d4-b8ef49c2bc95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:47.829329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829320    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829379    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829411    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829511 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829476    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829571    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829656    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.829768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.829729    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.830226 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.830205    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:47.830335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.830256    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.831156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.831131    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z" pod "417e1195-64de-4597-96d4-b8ef49c2bc95" (UID: "417e1195-64de-4597-96d4-b8ef49c2bc95"). InnerVolumeSpecName "kube-api-access-qfn6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:47.832035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.832008    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca" pod "417e1195-64de-4597-96d4-b8ef49c2bc95" (UID: "417e1195-64de-4597-96d4-b8ef49c2bc95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:47.832149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.832130    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.832292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.832276    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.844038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.843998    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xw8\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.844246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.844223    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:47.932080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.932044    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417e1195-64de-4597-96d4-b8ef49c2bc95-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:47.932080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.932078    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfn6z\" (UniqueName: \"kubernetes.io/projected/417e1195-64de-4597-96d4-b8ef49c2bc95-kube-api-access-qfn6z\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:47.936873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:47.936846    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:48.236340 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:48.236310    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa62ff0f_0bdf_4d23_82a9_7ff8b04075c2.slice/crio-dd2390dbda662dfc462221ebb324bf19c6c0ee3208ce8b6e55ac9b897b6e521b WatchSource:0}: Error finding container dd2390dbda662dfc462221ebb324bf19c6c0ee3208ce8b6e55ac9b897b6e521b: Status 404 returned error can't find the container with id dd2390dbda662dfc462221ebb324bf19c6c0ee3208ce8b6e55ac9b897b6e521b
Nov 03 23:44:48.237296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.237228    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"]
Nov 03 23:44:48.239202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.239174    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:48.239299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.239242    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:48.239299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.239274    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:44:48.597925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.597834    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"
Nov 03 23:44:48.597925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.597862    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t" event={"ID":"417e1195-64de-4597-96d4-b8ef49c2bc95","Type":"ContainerDied","Data":"857cf5474c318ba45c3469fa17723bc4f9fdde1456eff53b3bb6abac0734b44e"}
Nov 03 23:44:48.598137 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.597927    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="857cf5474c318ba45c3469fa17723bc4f9fdde1456eff53b3bb6abac0734b44e"
Nov 03 23:44:48.599987 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.599960    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" event={"ID":"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2","Type":"ContainerStarted","Data":"ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa"}
Nov 03 23:44:48.600153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.600009    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" event={"ID":"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2","Type":"ContainerStarted","Data":"dd2390dbda662dfc462221ebb324bf19c6c0ee3208ce8b6e55ac9b897b6e521b"}
Nov 03 23:44:48.633616 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.633553    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podStartSLOduration=1.633535272 podStartE2EDuration="1.633535272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:48.629296265 +0000 UTC m=+953.728896410" watchObservedRunningTime="2026-11-03 23:44:48.633535272 +0000 UTC m=+953.733135416"
Nov 03 23:44:48.937557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:48.937517    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:49.938859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:49.938821    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 23:44:49.939259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:49.938884    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 23:44:50.625763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.625048    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"]
Nov 03 23:44:50.628758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.625972    5074 cpu_manager.go:414] "containerMap: removing container" podUID="417e1195-64de-4597-96d4-b8ef49c2bc95" containerName="revoke-keys"
Nov 03 23:44:50.628758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.625997    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417e1195-64de-4597-96d4-b8ef49c2bc95" containerName="revoke-keys"
Nov 03 23:44:50.628758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.626181    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="417e1195-64de-4597-96d4-b8ef49c2bc95" containerName="revoke-keys"
Nov 03 23:44:50.632046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.632021    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.634281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.634260    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"]
Nov 03 23:44:50.639310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.639282    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:44:50.640978 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.640950    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:44:50.673836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.673787    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.673999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.673972    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.674308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.674285    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccm7g\" (UniqueName: \"kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.775824 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.775786    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccm7g\" (UniqueName: \"kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.776006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.775894    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.776006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.775945    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.776709 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.776686    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.778328 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.778293    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.790991 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.790968    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccm7g\" (UniqueName: \"kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g\") pod \"maas-api-revoke-sub-3a4817a2-qmnl4\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:50.937451 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.937405    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 23:44:50.937617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.937489    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 23:44:50.953213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:50.953178    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:51.279261 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:51.279228    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"]
Nov 03 23:44:51.279917 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:51.279893    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f92671_f4b0_4fd0_a5de_fd6a273cd295.slice/crio-72f7e4a4c865eb2a78cf2f2b1255c14222e2b18805c30c6157a2345a6cc7c790 WatchSource:0}: Error finding container 72f7e4a4c865eb2a78cf2f2b1255c14222e2b18805c30c6157a2345a6cc7c790: Status 404 returned error can't find the container with id 72f7e4a4c865eb2a78cf2f2b1255c14222e2b18805c30c6157a2345a6cc7c790
Nov 03 23:44:51.669980 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:51.669943    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4" event={"ID":"97f92671-f4b0-4fd0-a5de-fd6a273cd295","Type":"ContainerStarted","Data":"72f7e4a4c865eb2a78cf2f2b1255c14222e2b18805c30c6157a2345a6cc7c790"}
Nov 03 23:44:51.937907 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:51.937868    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 23:44:51.938105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:51.937939    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 23:44:52.680777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:52.680674    5074 generic.go:356] "Generic (PLEG): container finished" podID="97f92671-f4b0-4fd0-a5de-fd6a273cd295" containerID="864391cd090f6ebc69ab6b059f59f6a36c884f28f3522e63f578b6071d390b5e" exitCode=0
Nov 03 23:44:52.681246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:52.680785    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4" event={"ID":"97f92671-f4b0-4fd0-a5de-fd6a273cd295","Type":"ContainerDied","Data":"864391cd090f6ebc69ab6b059f59f6a36c884f28f3522e63f578b6071d390b5e"}
Nov 03 23:44:52.941325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:52.941291    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 23:44:52.942510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:52.941581    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 23:44:53.767861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.767830    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"]
Nov 03 23:44:53.772529 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.772510    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.780240 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.780221    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"]
Nov 03 23:44:53.843810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.843421    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.843810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.843579    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.843810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.843627    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnj5\" (UniqueName: \"kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.943158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.942793    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 23:44:53.943158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.942857    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 23:44:53.946921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.946723    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnj5\" (UniqueName: \"kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.947765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.947119    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.949252 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.947930    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.949252 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.948999    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.970900 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.964170    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:53.978780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:53.976344    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnj5\" (UniqueName: \"kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5\") pod \"maas-api-revoke-sub-d9e19ddf-9xc2l\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:54.044679 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.044657    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:54.060414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.060383    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"]
Nov 03 23:44:54.060990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.060977    5074 cpu_manager.go:414] "containerMap: removing container" podUID="97f92671-f4b0-4fd0-a5de-fd6a273cd295" containerName="revoke-keys"
Nov 03 23:44:54.061049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.060991    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f92671-f4b0-4fd0-a5de-fd6a273cd295" containerName="revoke-keys"
Nov 03 23:44:54.061120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.061111    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f92671-f4b0-4fd0-a5de-fd6a273cd295" containerName="revoke-keys"
Nov 03 23:44:54.064474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.064458    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.078042 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.078014    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"]
Nov 03 23:44:54.142806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.142774    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:54.150268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.150050    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g\" (UniqueName: \"kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g\") pod \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") "
Nov 03 23:44:54.150268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.150195    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca\") pod \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") "
Nov 03 23:44:54.150402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.150333    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca\") pod \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\" (UID: \"97f92671-f4b0-4fd0-a5de-fd6a273cd295\") "
Nov 03 23:44:54.150789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.150761    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n57m\" (UniqueName: \"kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.150942 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.150889    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.151128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.151054    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.152124 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.152098    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca" pod "97f92671-f4b0-4fd0-a5de-fd6a273cd295" (UID: "97f92671-f4b0-4fd0-a5de-fd6a273cd295"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:54.154783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.154710    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca" pod "97f92671-f4b0-4fd0-a5de-fd6a273cd295" (UID: "97f92671-f4b0-4fd0-a5de-fd6a273cd295"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:54.156357 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.155828    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g" pod "97f92671-f4b0-4fd0-a5de-fd6a273cd295" (UID: "97f92671-f4b0-4fd0-a5de-fd6a273cd295"). InnerVolumeSpecName "kube-api-access-ccm7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:54.252717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.252681    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.252879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.252865    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n57m\" (UniqueName: \"kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.252972 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.252942    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.253043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.253020    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:54.253105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.253041    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f92671-f4b0-4fd0-a5de-fd6a273cd295-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:54.253105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.253059    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccm7g\" (UniqueName: \"kubernetes.io/projected/97f92671-f4b0-4fd0-a5de-fd6a273cd295-kube-api-access-ccm7g\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:54.254628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.254432    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.259005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.258977    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.268699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.268644    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n57m\" (UniqueName: \"kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m\") pod \"maas-api-revoke-sub-279ff644-dk9kw\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.387160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.387127    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:54.456791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.456766    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"]
Nov 03 23:44:54.458998 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:54.458936    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7801f3b4_01cb_42f2_99d9_71d01cec1d40.slice/crio-e60582acdc3e69c75c57bfbfcc8452bce4f37d855b85f3b8ad848e57028369b9 WatchSource:0}: Error finding container e60582acdc3e69c75c57bfbfcc8452bce4f37d855b85f3b8ad848e57028369b9: Status 404 returned error can't find the container with id e60582acdc3e69c75c57bfbfcc8452bce4f37d855b85f3b8ad848e57028369b9
Nov 03 23:44:54.692083 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:54.692043    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa992b3_7a8d_4217_8e38_cf9ae79857bf.slice/crio-5810a0089d73291cca0b4db54af62fa4897c4f4a64b032c6f087aaa2d98f21de WatchSource:0}: Error finding container 5810a0089d73291cca0b4db54af62fa4897c4f4a64b032c6f087aaa2d98f21de: Status 404 returned error can't find the container with id 5810a0089d73291cca0b4db54af62fa4897c4f4a64b032c6f087aaa2d98f21de
Nov 03 23:44:54.692764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.692656    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"]
Nov 03 23:44:54.702779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.699585    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"
Nov 03 23:44:54.702779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.699646    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4" event={"ID":"97f92671-f4b0-4fd0-a5de-fd6a273cd295","Type":"ContainerDied","Data":"72f7e4a4c865eb2a78cf2f2b1255c14222e2b18805c30c6157a2345a6cc7c790"}
Nov 03 23:44:54.702779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.699695    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f7e4a4c865eb2a78cf2f2b1255c14222e2b18805c30c6157a2345a6cc7c790"
Nov 03 23:44:54.702779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.701606    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l" event={"ID":"7801f3b4-01cb-42f2-99d9-71d01cec1d40","Type":"ContainerStarted","Data":"e60582acdc3e69c75c57bfbfcc8452bce4f37d855b85f3b8ad848e57028369b9"}
Nov 03 23:44:54.938564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.938533    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 23:44:54.938943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:54.938586    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 23:44:55.717208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.717169    5074 generic.go:356] "Generic (PLEG): container finished" podID="7801f3b4-01cb-42f2-99d9-71d01cec1d40" containerID="715286f7a99e0968884e57d4f3b0b58bf5ad02e14d56871eb7eb28bb3e973df1" exitCode=0
Nov 03 23:44:55.717480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.717456    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l" event={"ID":"7801f3b4-01cb-42f2-99d9-71d01cec1d40","Type":"ContainerDied","Data":"715286f7a99e0968884e57d4f3b0b58bf5ad02e14d56871eb7eb28bb3e973df1"}
Nov 03 23:44:55.719907 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.719878    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw" event={"ID":"3fa992b3-7a8d-4217-8e38-cf9ae79857bf","Type":"ContainerStarted","Data":"d57e2550bffed2d0e79845b61ad029f508427e84bd0e97c03dcd184165af12c2"}
Nov 03 23:44:55.720024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.719932    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw" event={"ID":"3fa992b3-7a8d-4217-8e38-cf9ae79857bf","Type":"ContainerStarted","Data":"5810a0089d73291cca0b4db54af62fa4897c4f4a64b032c6f087aaa2d98f21de"}
Nov 03 23:44:55.757810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.757763    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw" podStartSLOduration=1.75775082 podStartE2EDuration="1.75775082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:55.756340224 +0000 UTC m=+960.855940370" watchObservedRunningTime="2026-11-03 23:44:55.75775082 +0000 UTC m=+960.857350956"
Nov 03 23:44:55.946999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.946967    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:55.947498 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.947383    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:55.948609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:55.948586    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 03 23:44:56.742017 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:56.741977    5074 generic.go:356] "Generic (PLEG): container finished" podID="3fa992b3-7a8d-4217-8e38-cf9ae79857bf" containerID="d57e2550bffed2d0e79845b61ad029f508427e84bd0e97c03dcd184165af12c2" exitCode=0
Nov 03 23:44:56.742525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:56.742451    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw" event={"ID":"3fa992b3-7a8d-4217-8e38-cf9ae79857bf","Type":"ContainerDied","Data":"d57e2550bffed2d0e79845b61ad029f508427e84bd0e97c03dcd184165af12c2"}
Nov 03 23:44:56.943722 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:56.943694    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:57.028016 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.027937    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"]
Nov 03 23:44:57.028588 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.028567    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7801f3b4-01cb-42f2-99d9-71d01cec1d40" containerName="revoke-keys"
Nov 03 23:44:57.028588 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.028587    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7801f3b4-01cb-42f2-99d9-71d01cec1d40" containerName="revoke-keys"
Nov 03 23:44:57.028787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.028689    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7801f3b4-01cb-42f2-99d9-71d01cec1d40" containerName="revoke-keys"
Nov 03 23:44:57.032224 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.032203    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.040562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.040530    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 23:44:57.040654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.040595    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 23:44:57.040654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.040637    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 23:44:57.041507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.041326    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q44ht\""
Nov 03 23:44:57.047592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.047562    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"]
Nov 03 23:44:57.088873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.088842    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5\" (UniqueName: \"kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5\") pod \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") "
Nov 03 23:44:57.089044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.088949    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca\") pod \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") "
Nov 03 23:44:57.089044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.088988    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca\") pod \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\" (UID: \"7801f3b4-01cb-42f2-99d9-71d01cec1d40\") "
Nov 03 23:44:57.089283 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.089261    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.089363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.089313    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.089400 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.089353    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca" pod "7801f3b4-01cb-42f2-99d9-71d01cec1d40" (UID: "7801f3b4-01cb-42f2-99d9-71d01cec1d40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:57.089434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.089399    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86wl\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.089989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.089698    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.089989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.089787    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.090174 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.090022    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.090174 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.090059    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.090248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.090178    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:57.091306 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.091279    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca" pod "7801f3b4-01cb-42f2-99d9-71d01cec1d40" (UID: "7801f3b4-01cb-42f2-99d9-71d01cec1d40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:57.091306 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.091280    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5" pod "7801f3b4-01cb-42f2-99d9-71d01cec1d40" (UID: "7801f3b4-01cb-42f2-99d9-71d01cec1d40"). InnerVolumeSpecName "kube-api-access-cbnj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:57.191007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.190968    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191021    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191101    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191125    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191185    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191429 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191308    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191429 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191363    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86wl\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.191577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191559    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbnj5\" (UniqueName: \"kubernetes.io/projected/7801f3b4-01cb-42f2-99d9-71d01cec1d40-kube-api-access-cbnj5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:57.191647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191584    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7801f3b4-01cb-42f2-99d9-71d01cec1d40-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:57.191967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.191943    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.193597 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.193570    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.193699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.193641    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.193904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.193886    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.194039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.194021    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.203985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.203962    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.204777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.204731    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86wl\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-497vs\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.347773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.347675    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:44:57.638703 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.638670    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"]
Nov 03 23:44:57.638860 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:44:57.638715    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f9b005_a880_4895_8b97_b9d59aec49b4.slice/crio-5d62f413df7823c024fab7814a5287c8459332ab6428545456545a48f8ec089a WatchSource:0}: Error finding container 5d62f413df7823c024fab7814a5287c8459332ab6428545456545a48f8ec089a: Status 404 returned error can't find the container with id 5d62f413df7823c024fab7814a5287c8459332ab6428545456545a48f8ec089a
Nov 03 23:44:57.641297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.641281    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:44:57.751443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.751415    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"
Nov 03 23:44:57.751617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.751414    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l" event={"ID":"7801f3b4-01cb-42f2-99d9-71d01cec1d40","Type":"ContainerDied","Data":"e60582acdc3e69c75c57bfbfcc8452bce4f37d855b85f3b8ad848e57028369b9"}
Nov 03 23:44:57.751617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.751540    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60582acdc3e69c75c57bfbfcc8452bce4f37d855b85f3b8ad848e57028369b9"
Nov 03 23:44:57.752785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.752761    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" event={"ID":"89f9b005-a880-4895-8b97-b9d59aec49b4","Type":"ContainerStarted","Data":"5d62f413df7823c024fab7814a5287c8459332ab6428545456545a48f8ec089a"}
Nov 03 23:44:57.898108 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:57.898085    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:58.002068 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.002037    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca\") pod \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") "
Nov 03 23:44:58.002226 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.002107    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m\" (UniqueName: \"kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m\") pod \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") "
Nov 03 23:44:58.002336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.002314    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca\") pod \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\" (UID: \"3fa992b3-7a8d-4217-8e38-cf9ae79857bf\") "
Nov 03 23:44:58.002864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.002812    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca" pod "3fa992b3-7a8d-4217-8e38-cf9ae79857bf" (UID: "3fa992b3-7a8d-4217-8e38-cf9ae79857bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:44:58.003133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.003095    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:58.005216 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.005183    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca" pod "3fa992b3-7a8d-4217-8e38-cf9ae79857bf" (UID: "3fa992b3-7a8d-4217-8e38-cf9ae79857bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:44:58.005562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.005541    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m" pod "3fa992b3-7a8d-4217-8e38-cf9ae79857bf" (UID: "3fa992b3-7a8d-4217-8e38-cf9ae79857bf"). InnerVolumeSpecName "kube-api-access-2n57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:44:58.030814 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:44:58.030708    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:44:58.104280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.104240    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:58.104280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.104273    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n57m\" (UniqueName: \"kubernetes.io/projected/3fa992b3-7a8d-4217-8e38-cf9ae79857bf-kube-api-access-2n57m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:44:58.762049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.762016    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"
Nov 03 23:44:58.762225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.762069    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw" event={"ID":"3fa992b3-7a8d-4217-8e38-cf9ae79857bf","Type":"ContainerDied","Data":"5810a0089d73291cca0b4db54af62fa4897c4f4a64b032c6f087aaa2d98f21de"}
Nov 03 23:44:58.762225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:58.762129    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5810a0089d73291cca0b4db54af62fa4897c4f4a64b032c6f087aaa2d98f21de"
Nov 03 23:44:59.283012 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.282975    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"]
Nov 03 23:44:59.284469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.284442    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3fa992b3-7a8d-4217-8e38-cf9ae79857bf" containerName="revoke-keys"
Nov 03 23:44:59.284633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.284618    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa992b3-7a8d-4217-8e38-cf9ae79857bf" containerName="revoke-keys"
Nov 03 23:44:59.284980 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.284963    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa992b3-7a8d-4217-8e38-cf9ae79857bf" containerName="revoke-keys"
Nov 03 23:44:59.290062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.290039    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.296850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.296698    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f6f292-openshift-default-dockercfg-b884k\""
Nov 03 23:44:59.309816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.305265    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"]
Nov 03 23:44:59.423005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.422962    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423096    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423257 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423187    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pcd\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423320 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423287    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423522    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423565    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423721    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.423854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.423789    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541572    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541680    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pcd\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541720    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541797    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541837    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541865    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541966    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.541996    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.542099    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.542614    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.543620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.542903    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.550092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.545087    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.550092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.545317    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.550092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.545825    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.550092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.550042    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.563659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.563586    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.567405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.567351    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pcd\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.569852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.569798    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:44:59.623170 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:44:59.623132    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:45:00.369762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.315680    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"]
Nov 03 23:45:00.410195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.410160    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:00.415893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.415863    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:45:00.537755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.537016    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:00.537755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.537272    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mcm9\" (UniqueName: \"kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:00.596844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.574546    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"]
Nov 03 23:45:00.612906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.606024    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"]
Nov 03 23:45:00.637347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.629680    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:00.637347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.629783    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:00.637347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.629826    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:00.637347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.630838    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:00.653760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.652594    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlzq\" (UniqueName: \"kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:00.653760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.652659    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:00.653760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.652697    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:00.653760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.652900    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mcm9\" (UniqueName: \"kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:00.692231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.691163    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:00.705022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.696231    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"]
Nov 03 23:45:00.705022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.696279    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"]
Nov 03 23:45:00.705022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.696377    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:00.707658 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.707575    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"]
Nov 03 23:45:00.788575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.758766    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlzq\" (UniqueName: \"kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:00.788575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.758840    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:00.788575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.759808    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:00.852844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.852808    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"]
Nov 03 23:45:00.916812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.912828    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895825-bjwls\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:00.916812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.912907    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv42l\" (UniqueName: \"kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l\") pod \"maas-api-key-cleanup-29895825-bjwls\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:00.947148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.944581    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"]
Nov 03 23:45:00.947148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.944629    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"]
Nov 03 23:45:00.947148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:00.944728    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.049765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.022968    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895825-bjwls\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:01.049765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.023043    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv42l\" (UniqueName: \"kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l\") pod \"maas-api-key-cleanup-29895825-bjwls\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:01.049765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.024855    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895825-bjwls\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:01.084766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.063833    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mcm9\" (UniqueName: \"kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:01.099061 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.097237    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlzq\" (UniqueName: \"kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:01.141632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.141273    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbpqv\" (UniqueName: \"kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.141632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.141428    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.306781 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.306486    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:01.330989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.326941    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.330989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.327261    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpqv\" (UniqueName: \"kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.330989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.328780    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.373084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.347566    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:01.519658 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.518599    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv42l\" (UniqueName: \"kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l\") pod \"maas-api-key-cleanup-29895825-bjwls\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:01.555764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.530572    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:01.628581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.606622    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpqv\" (UniqueName: \"kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.628581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.625130    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" event={"ID":"951a1a46-d504-4b34-9f6b-12d2a57ba342","Type":"ContainerStarted","Data":"ed0be28b0ec1090c8e5c4c64710683586b0f7d1387974a9ac8bee3324e86e582"}
Nov 03 23:45:01.628581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.625197    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"]
Nov 03 23:45:01.634176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.634151    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"]
Nov 03 23:45:01.634466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.634452    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720431    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720511    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720566    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstlh\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720603    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720649    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720801    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720851    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720950    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.734160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.720974    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.796564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.783920    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:01.822777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.822655    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.822777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.822706    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.822777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.822774    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.823067 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.822834    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.823067 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.822888    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstlh\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.823067 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.822922    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.836769    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.836989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.837055    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.838106    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.838371    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.838625    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.841390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.841365    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.867339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.841891    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.979699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.942065    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.979699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.959269    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.979699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.979128    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:01.995581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:01.992720    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstlh\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:02.247297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:02.246991    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:02.899780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:02.785161    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.906891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:02.906849    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.041823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.013875    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.425932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.420604    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"]
Nov 03 23:45:03.425932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.420653    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" event={"ID":"951a1a46-d504-4b34-9f6b-12d2a57ba342","Type":"ContainerStarted","Data":"0fabca77f932d902cb63f9adbc9c1c5f73a51001be06db8477f3467ecf4a9601"}
Nov 03 23:45:03.600863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.583788    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podStartSLOduration=4.502722723 podStartE2EDuration="4.502722723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:03.441082387 +0000 UTC m=+968.540682531" watchObservedRunningTime="2026-11-03 23:45:03.502722723 +0000 UTC m=+968.602322868"
Nov 03 23:45:03.629037 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:45:03.620852    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce657cd_200f_494a_b6dc_9005e58e93ae.slice/crio-ea6641f2842cdb4a2eb814e59faa2ded69dc400ffa90b06a5be501cddb3f3532 WatchSource:0}: Error finding container ea6641f2842cdb4a2eb814e59faa2ded69dc400ffa90b06a5be501cddb3f3532: Status 404 returned error can't find the container with id ea6641f2842cdb4a2eb814e59faa2ded69dc400ffa90b06a5be501cddb3f3532
Nov 03 23:45:03.631470 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.631423    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:45:03.685389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.685356    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:03.712239 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.712204    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:03.957141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:03.954620    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"]
Nov 03 23:45:04.127801 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.115807    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"]
Nov 03 23:45:04.168833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.156827    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:04.168833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.156882    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.196153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.192399    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"]
Nov 03 23:45:04.430685 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.414022    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.461141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.448062    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.738888 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.738854    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:04.740771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.740718    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:04.965873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:04.965784    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerStarted","Data":"710765f0af42f7164ca5f5639309b3aec9c1f7de81878bd42cc48a8f2e0b1fa4"}
Nov 03 23:45:05.429780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.429646    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:05.634878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.604917    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:45:05.634878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.604974    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerStarted","Data":"803640adff49cfe9f7c53d1ba4566216616889aa36d39cc9d8cd5ce9e45ca8a5"}
Nov 03 23:45:05.634878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.605109    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" event={"ID":"3e30a80b-6eaf-4b13-abda-130da48dbf3e","Type":"ContainerStarted","Data":"a081b45235fe424671f44be81a2f5315d3938bfc0dabf4e8a8ac936fd1b35af5"}
Nov 03 23:45:05.634878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.605156    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerStarted","Data":"ea6641f2842cdb4a2eb814e59faa2ded69dc400ffa90b06a5be501cddb3f3532"}
Nov 03 23:45:05.712763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.711987    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:05.712763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.712045    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:05.769936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.718350    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:05.769936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.718496    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:05.769936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.718594    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:05.769936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.720241    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:05.769936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.733332    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:05.794763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.787049    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"]
Nov 03 23:45:05.992155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:05.992116    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:06.046157 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.040194    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:45:06.475629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.475594    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerStarted","Data":"3070d7d2e08516a5cf9be62676d70fd12c3542a5d315e231f73206c349c7d7ab"}
Nov 03 23:45:06.479419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.479392    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerStarted","Data":"cd65e6c2bdbfc54cf9caec3c67b07e07689591f1fa4c5d70baf8b55917680123"}
Nov 03 23:45:06.483467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.483442    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerStarted","Data":"cdbaca8ae075e9b2e9d3aef328e7373d7c6f0851a38dbe5a1d12b7c16f5129c2"}
Nov 03 23:45:06.486251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.486221    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" event={"ID":"347a2c82-670c-455f-b442-482ccb9e7fa1","Type":"ContainerStarted","Data":"afacac5a549750daf027f13d1bc78dfd23e5799abdab8bca8f5b5b6dc84f11b3"}
Nov 03 23:45:06.552141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.552077    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" podStartSLOduration=6.552054512 podStartE2EDuration="6.552054512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:06.53868985 +0000 UTC m=+971.638289995" watchObservedRunningTime="2026-11-03 23:45:06.552054512 +0000 UTC m=+971.651654657"
Nov 03 23:45:06.575173 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.575116    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" podStartSLOduration=6.575066535 podStartE2EDuration="6.575066535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:06.571726421 +0000 UTC m=+971.671326563" watchObservedRunningTime="2026-11-03 23:45:06.575066535 +0000 UTC m=+971.674666678"
Nov 03 23:45:06.594990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.594938    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" podStartSLOduration=6.594923609 podStartE2EDuration="6.594923609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:06.59286796 +0000 UTC m=+971.692468102" watchObservedRunningTime="2026-11-03 23:45:06.594923609 +0000 UTC m=+971.694523752"
Nov 03 23:45:06.632449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.632013    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:06.632449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:06.632073    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:07.258342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.258292    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"]
Nov 03 23:45:07.266070 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.266038    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.268118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.267866    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"]
Nov 03 23:45:07.272003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.271981    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:45:07.281177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.281153    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.281271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.281201    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.281424 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.281407    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxs7\" (UniqueName: \"kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.382948 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.382913    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.383114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.382970    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.383114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.383106    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxs7\" (UniqueName: \"kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.383784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.383732    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.386153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.386129    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.417999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.417630    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxs7\" (UniqueName: \"kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7\") pod \"maas-api-revoke-sub-2d0b6e31-j2rgk\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.501770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.499274    5074 generic.go:356] "Generic (PLEG): container finished" podID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerID="88e2e539e1abd84beb68699c84fb3e2e3f136ecb676cad91841468d5df3d7b9f" exitCode=60
Nov 03 23:45:07.501770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.499399    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" event={"ID":"3e30a80b-6eaf-4b13-abda-130da48dbf3e","Type":"ContainerDied","Data":"88e2e539e1abd84beb68699c84fb3e2e3f136ecb676cad91841468d5df3d7b9f"}
Nov 03 23:45:07.516823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.505879    5074 scope.go:122] "RemoveContainer" containerID="88e2e539e1abd84beb68699c84fb3e2e3f136ecb676cad91841468d5df3d7b9f"
Nov 03 23:45:07.568787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.565213    5074 generic.go:356] "Generic (PLEG): container finished" podID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerID="3070d7d2e08516a5cf9be62676d70fd12c3542a5d315e231f73206c349c7d7ab" exitCode=60
Nov 03 23:45:07.568787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.565325    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerDied","Data":"3070d7d2e08516a5cf9be62676d70fd12c3542a5d315e231f73206c349c7d7ab"}
Nov 03 23:45:07.568787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.565805    5074 scope.go:122] "RemoveContainer" containerID="3070d7d2e08516a5cf9be62676d70fd12c3542a5d315e231f73206c349c7d7ab"
Nov 03 23:45:07.570245 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.569262    5074 generic.go:356] "Generic (PLEG): container finished" podID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerID="cd65e6c2bdbfc54cf9caec3c67b07e07689591f1fa4c5d70baf8b55917680123" exitCode=60
Nov 03 23:45:07.570245 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.569799    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerDied","Data":"cd65e6c2bdbfc54cf9caec3c67b07e07689591f1fa4c5d70baf8b55917680123"}
Nov 03 23:45:07.570754 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.570573    5074 scope.go:122] "RemoveContainer" containerID="cd65e6c2bdbfc54cf9caec3c67b07e07689591f1fa4c5d70baf8b55917680123"
Nov 03 23:45:07.577235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.577215    5074 generic.go:356] "Generic (PLEG): container finished" podID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerID="cdbaca8ae075e9b2e9d3aef328e7373d7c6f0851a38dbe5a1d12b7c16f5129c2" exitCode=60
Nov 03 23:45:07.577537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.577516    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerDied","Data":"cdbaca8ae075e9b2e9d3aef328e7373d7c6f0851a38dbe5a1d12b7c16f5129c2"}
Nov 03 23:45:07.578056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.578038    5074 scope.go:122] "RemoveContainer" containerID="cdbaca8ae075e9b2e9d3aef328e7373d7c6f0851a38dbe5a1d12b7c16f5129c2"
Nov 03 23:45:07.582014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.581993    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:07.585972 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.585945    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" event={"ID":"347a2c82-670c-455f-b442-482ccb9e7fa1","Type":"ContainerStarted","Data":"ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6"}
Nov 03 23:45:07.638397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.633859    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:07.638397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.633934    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:07.671020 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:07.655192    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podStartSLOduration=7.655172061 podStartE2EDuration="7.655172061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:07.647478709 +0000 UTC m=+972.747078891" watchObservedRunningTime="2026-11-03 23:45:07.655172061 +0000 UTC m=+972.754772206"
Nov 03 23:45:08.273878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.273792    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:08.282043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.281480    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 23:45:08.282043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.281525    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 23:45:08.284891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.284866    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"]
Nov 03 23:45:08.605769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.597782    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerStarted","Data":"556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86"}
Nov 03 23:45:08.621763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.608023    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk" event={"ID":"d3f16b6b-5302-48c0-819b-757f3e34ea20","Type":"ContainerStarted","Data":"38594a45f9a3653161d53dbed37348390453ab4c1983bf0a44b2f7cae83ce878"}
Nov 03 23:45:08.624225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.623836    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:08.624225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.623890    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:08.677772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.677226    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" event={"ID":"3e30a80b-6eaf-4b13-abda-130da48dbf3e","Type":"ContainerStarted","Data":"0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0"}
Nov 03 23:45:08.684830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.684799    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerStarted","Data":"398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5"}
Nov 03 23:45:08.687854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.687556    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerStarted","Data":"990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e"}
Nov 03 23:45:08.706787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:08.701061    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" podStartSLOduration=8.701045004000001 podStartE2EDuration="8.701045004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:08.698016193 +0000 UTC m=+973.797616338" watchObservedRunningTime="2026-11-03 23:45:08.701045004 +0000 UTC m=+973.800645149"
Nov 03 23:45:09.258573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.258475    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 23:45:09.258897 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.258845    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 23:45:09.625092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.624527    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:09.625092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.624591    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:09.758608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.740011    5074 generic.go:356] "Generic (PLEG): container finished" podID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerID="556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86" exitCode=60
Nov 03 23:45:09.758608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.740187    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerDied","Data":"556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86"}
Nov 03 23:45:09.758608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.740242    5074 scope.go:122] "RemoveContainer" containerID="cdbaca8ae075e9b2e9d3aef328e7373d7c6f0851a38dbe5a1d12b7c16f5129c2"
Nov 03 23:45:09.758608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.740760    5074 scope.go:122] "RemoveContainer" containerID="556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86"
Nov 03 23:45:09.758608 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:09.741145    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg_odh-ai-gateway-infra(5c14555e-0ccb-4ee5-8e38-76fa553b6b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b"
Nov 03 23:45:09.792884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.792842    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk" event={"ID":"d3f16b6b-5302-48c0-819b-757f3e34ea20","Type":"ContainerStarted","Data":"a02eb76007ac116e8eb25c654ac263c162d2e1e03e32eca1b1fdbdc7d437b79a"}
Nov 03 23:45:09.797342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.797303    5074 generic.go:356] "Generic (PLEG): container finished" podID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerID="0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0" exitCode=60
Nov 03 23:45:09.797642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.797615    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" event={"ID":"3e30a80b-6eaf-4b13-abda-130da48dbf3e","Type":"ContainerDied","Data":"0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0"}
Nov 03 23:45:09.809814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.809789    5074 scope.go:122] "RemoveContainer" containerID="0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0"
Nov 03 23:45:09.810678 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:09.810650    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2_odh-ai-gateway-infra(3e30a80b-6eaf-4b13-abda-130da48dbf3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e"
Nov 03 23:45:09.823459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.823433    5074 generic.go:356] "Generic (PLEG): container finished" podID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerID="398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5" exitCode=60
Nov 03 23:45:09.829884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.829855    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerDied","Data":"398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5"}
Nov 03 23:45:09.830455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.830436    5074 scope.go:122] "RemoveContainer" containerID="398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5"
Nov 03 23:45:09.838929 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:09.830880    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc_odh-ai-gateway-infra(2ce657cd-200f-494a-b6dc-9005e58e93ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae"
Nov 03 23:45:09.870047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.868288    5074 generic.go:356] "Generic (PLEG): container finished" podID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerID="990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e" exitCode=60
Nov 03 23:45:09.870047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.869535    5074 scope.go:122] "RemoveContainer" containerID="990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e"
Nov 03 23:45:09.870047 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:09.869836    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895825-bjwls_odh-ai-gateway-infra(9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6"
Nov 03 23:45:09.870047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.869885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerDied","Data":"990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e"}
Nov 03 23:45:09.871342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.871230    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk" podStartSLOduration=2.871213507 podStartE2EDuration="2.871213507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:09.848755091 +0000 UTC m=+974.948355234" watchObservedRunningTime="2026-11-03 23:45:09.871213507 +0000 UTC m=+974.970813656"
Nov 03 23:45:09.897673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.897587    5074 scope.go:122] "RemoveContainer" containerID="88e2e539e1abd84beb68699c84fb3e2e3f136ecb676cad91841468d5df3d7b9f"
Nov 03 23:45:09.953752 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.953717    5074 scope.go:122] "RemoveContainer" containerID="3070d7d2e08516a5cf9be62676d70fd12c3542a5d315e231f73206c349c7d7ab"
Nov 03 23:45:09.975057 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:09.975032    5074 scope.go:122] "RemoveContainer" containerID="cd65e6c2bdbfc54cf9caec3c67b07e07689591f1fa4c5d70baf8b55917680123"
Nov 03 23:45:10.248940 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.248898    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 23:45:10.249119 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.248978    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 23:45:10.624625 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.624128    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:45:10.624625 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.624195    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 23:45:10.920824 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.918099    5074 scope.go:122] "RemoveContainer" containerID="0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0"
Nov 03 23:45:10.920824 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:10.918460    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2_odh-ai-gateway-infra(3e30a80b-6eaf-4b13-abda-130da48dbf3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e"
Nov 03 23:45:10.921371 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.921282    5074 scope.go:122] "RemoveContainer" containerID="398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5"
Nov 03 23:45:10.937764 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:10.921572    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc_odh-ai-gateway-infra(2ce657cd-200f-494a-b6dc-9005e58e93ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae"
Nov 03 23:45:10.945542 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.939635    5074 scope.go:122] "RemoveContainer" containerID="990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e"
Nov 03 23:45:10.945542 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:10.939989    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895825-bjwls_odh-ai-gateway-infra(9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6"
Nov 03 23:45:10.963611 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.963584    5074 scope.go:122] "RemoveContainer" containerID="556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86"
Nov 03 23:45:10.964534 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:10.964505    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg_odh-ai-gateway-infra(5c14555e-0ccb-4ee5-8e38-76fa553b6b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b"
Nov 03 23:45:10.966375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.966333    5074 generic.go:356] "Generic (PLEG): container finished" podID="d3f16b6b-5302-48c0-819b-757f3e34ea20" containerID="a02eb76007ac116e8eb25c654ac263c162d2e1e03e32eca1b1fdbdc7d437b79a" exitCode=0
Nov 03 23:45:10.966674 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:10.966534    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk" event={"ID":"d3f16b6b-5302-48c0-819b-757f3e34ea20","Type":"ContainerDied","Data":"a02eb76007ac116e8eb25c654ac263c162d2e1e03e32eca1b1fdbdc7d437b79a"}
Nov 03 23:45:11.252056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.252025    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 23:45:11.252267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.252243    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 23:45:11.869114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.869077    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:45:11.870610 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.870406    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:45:11.876690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.876667    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 03 23:45:11.946214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.945952    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:11.946214 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:11.946035    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:45:12.248662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.248632    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 23:45:12.248830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.248727    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 23:45:12.278878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.278596    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:12.387454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.387361    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7\" (UniqueName: \"kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7\") pod \"d3f16b6b-5302-48c0-819b-757f3e34ea20\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") "
Nov 03 23:45:12.387614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.387457    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca\") pod \"d3f16b6b-5302-48c0-819b-757f3e34ea20\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") "
Nov 03 23:45:12.387614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.387585    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca\") pod \"d3f16b6b-5302-48c0-819b-757f3e34ea20\" (UID: \"d3f16b6b-5302-48c0-819b-757f3e34ea20\") "
Nov 03 23:45:12.388231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.387911    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca" pod "d3f16b6b-5302-48c0-819b-757f3e34ea20" (UID: "d3f16b6b-5302-48c0-819b-757f3e34ea20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:12.388231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.388203    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:12.390308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.390260    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca" pod "d3f16b6b-5302-48c0-819b-757f3e34ea20" (UID: "d3f16b6b-5302-48c0-819b-757f3e34ea20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:45:12.390476 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.390455    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7" pod "d3f16b6b-5302-48c0-819b-757f3e34ea20" (UID: "d3f16b6b-5302-48c0-819b-757f3e34ea20"). InnerVolumeSpecName "kube-api-access-lfxs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:12.500761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.498538    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f16b6b-5302-48c0-819b-757f3e34ea20-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:12.500761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.498574    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxs7\" (UniqueName: \"kubernetes.io/projected/d3f16b6b-5302-48c0-819b-757f3e34ea20-kube-api-access-lfxs7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:12.999800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.997830    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk" event={"ID":"d3f16b6b-5302-48c0-819b-757f3e34ea20","Type":"ContainerDied","Data":"38594a45f9a3653161d53dbed37348390453ab4c1983bf0a44b2f7cae83ce878"}
Nov 03 23:45:12.999800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.997898    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38594a45f9a3653161d53dbed37348390453ab4c1983bf0a44b2f7cae83ce878"
Nov 03 23:45:12.999800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:12.998010    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"
Nov 03 23:45:13.008425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.007816    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" event={"ID":"89f9b005-a880-4895-8b97-b9d59aec49b4","Type":"ContainerStarted","Data":"d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059"}
Nov 03 23:45:13.008425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.008139    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:45:13.011334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.011254    5074 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-497vs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:45:13.011334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.011301    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" podUID="89f9b005-a880-4895-8b97-b9d59aec49b4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:45:13.034618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.034565    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" podStartSLOduration=1.731758711 podStartE2EDuration="16.034553346s" totalImagesPullingTime="14.302794635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:13.032118071 +0000 UTC m=+978.131718216" watchObservedRunningTime="2026-11-03 23:45:13.034553346 +0000 UTC m=+978.134153490"
Nov 03 23:45:13.248860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.248829    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 23:45:13.249019 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:13.248916    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 23:45:14.018735 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:14.018706    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 03 23:45:14.364389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:14.364310    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:14.368415 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:14.367863    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:14.370767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:14.370152    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 03 23:45:21.020880 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:21.020852    5074 scope.go:122] "RemoveContainer" containerID="398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5"
Nov 03 23:45:22.010460 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:22.010427    5074 scope.go:122] "RemoveContainer" containerID="0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0"
Nov 03 23:45:22.010624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:22.010567    5074 scope.go:122] "RemoveContainer" containerID="556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86"
Nov 03 23:45:22.097708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:22.097674    5074 generic.go:356] "Generic (PLEG): container finished" podID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerID="f459040becdebdaba0de5e373814c47ee48ed310d0464851b0a8696206ccc6e8" exitCode=60
Nov 03 23:45:22.098184 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:22.097830    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerDied","Data":"f459040becdebdaba0de5e373814c47ee48ed310d0464851b0a8696206ccc6e8"}
Nov 03 23:45:22.098184 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:22.097892    5074 scope.go:122] "RemoveContainer" containerID="398bc149d48b41020bdb87eeaaa06fb59930c427a4ef417ce9e29f21df2d3cc5"
Nov 03 23:45:22.103897 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:22.098379    5074 scope.go:122] "RemoveContainer" containerID="f459040becdebdaba0de5e373814c47ee48ed310d0464851b0a8696206ccc6e8"
Nov 03 23:45:22.103897 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:22.098713    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc_odh-ai-gateway-infra(2ce657cd-200f-494a-b6dc-9005e58e93ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae"
Nov 03 23:45:23.010576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.010550    5074 scope.go:122] "RemoveContainer" containerID="990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e"
Nov 03 23:45:23.113479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.113451    5074 generic.go:356] "Generic (PLEG): container finished" podID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerID="d162c6b7a7e5dd1edc78bbe0eecf16d1362eaa29f1dcb3d003166966090649d0" exitCode=60
Nov 03 23:45:23.113930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.113530    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" event={"ID":"3e30a80b-6eaf-4b13-abda-130da48dbf3e","Type":"ContainerDied","Data":"d162c6b7a7e5dd1edc78bbe0eecf16d1362eaa29f1dcb3d003166966090649d0"}
Nov 03 23:45:23.113930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.113593    5074 scope.go:122] "RemoveContainer" containerID="0e07be3219e508bf777c8230afb919510378693e59ed397109c9d569e6d6caa0"
Nov 03 23:45:23.114101 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.114085    5074 scope.go:122] "RemoveContainer" containerID="d162c6b7a7e5dd1edc78bbe0eecf16d1362eaa29f1dcb3d003166966090649d0"
Nov 03 23:45:23.114385 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:23.114364    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2_odh-ai-gateway-infra(3e30a80b-6eaf-4b13-abda-130da48dbf3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e"
Nov 03 23:45:23.118249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.118224    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerStarted","Data":"44d5bda9f55daa9a8fee23ae298d581b223799beb0483983b25dc21f0de0afd4"}
Nov 03 23:45:23.118638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.118617    5074 scope.go:122] "RemoveContainer" containerID="44d5bda9f55daa9a8fee23ae298d581b223799beb0483983b25dc21f0de0afd4"
Nov 03 23:45:23.118942 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:23.118921    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg_odh-ai-gateway-infra(5c14555e-0ccb-4ee5-8e38-76fa553b6b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b"
Nov 03 23:45:23.137347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.137313    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"]
Nov 03 23:45:23.432380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.432344    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:23.612917 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.612837    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9\" (UniqueName: \"kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9\") pod \"2ce657cd-200f-494a-b6dc-9005e58e93ae\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") "
Nov 03 23:45:23.612917 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.612880    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca\") pod \"2ce657cd-200f-494a-b6dc-9005e58e93ae\" (UID: \"2ce657cd-200f-494a-b6dc-9005e58e93ae\") "
Nov 03 23:45:23.613279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.613255    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca" pod "2ce657cd-200f-494a-b6dc-9005e58e93ae" (UID: "2ce657cd-200f-494a-b6dc-9005e58e93ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:23.615082 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.615060    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9" pod "2ce657cd-200f-494a-b6dc-9005e58e93ae" (UID: "2ce657cd-200f-494a-b6dc-9005e58e93ae"). InnerVolumeSpecName "kube-api-access-9mcm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:23.717585 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.717552    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mcm9\" (UniqueName: \"kubernetes.io/projected/2ce657cd-200f-494a-b6dc-9005e58e93ae-kube-api-access-9mcm9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:23.717585 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:23.717584    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce657cd-200f-494a-b6dc-9005e58e93ae-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:24.132555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.132525    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"
Nov 03 23:45:24.132555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.132537    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc" event={"ID":"2ce657cd-200f-494a-b6dc-9005e58e93ae","Type":"ContainerDied","Data":"ea6641f2842cdb4a2eb814e59faa2ded69dc400ffa90b06a5be501cddb3f3532"}
Nov 03 23:45:24.133029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.132609    5074 scope.go:122] "RemoveContainer" containerID="f459040becdebdaba0de5e373814c47ee48ed310d0464851b0a8696206ccc6e8"
Nov 03 23:45:24.135198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.135175    5074 generic.go:356] "Generic (PLEG): container finished" podID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerID="a5f2e56d67687dd53abde576d214e90ea24eef2336035936cb854cb154e01e68" exitCode=60
Nov 03 23:45:24.135319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.135242    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerDied","Data":"a5f2e56d67687dd53abde576d214e90ea24eef2336035936cb854cb154e01e68"}
Nov 03 23:45:24.135727 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.135709    5074 scope.go:122] "RemoveContainer" containerID="a5f2e56d67687dd53abde576d214e90ea24eef2336035936cb854cb154e01e68"
Nov 03 23:45:24.136144 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:24.136121    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895825-bjwls_odh-ai-gateway-infra(9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6"
Nov 03 23:45:24.137512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.137493    5074 generic.go:356] "Generic (PLEG): container finished" podID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerID="44d5bda9f55daa9a8fee23ae298d581b223799beb0483983b25dc21f0de0afd4" exitCode=60
Nov 03 23:45:24.137602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.137548    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerDied","Data":"44d5bda9f55daa9a8fee23ae298d581b223799beb0483983b25dc21f0de0afd4"}
Nov 03 23:45:24.137899 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.137885    5074 scope.go:122] "RemoveContainer" containerID="44d5bda9f55daa9a8fee23ae298d581b223799beb0483983b25dc21f0de0afd4"
Nov 03 23:45:24.138126 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:24.138109    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg_odh-ai-gateway-infra(5c14555e-0ccb-4ee5-8e38-76fa553b6b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b"
Nov 03 23:45:24.148910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.148888    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"]
Nov 03 23:45:24.158797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.158774    5074 scope.go:122] "RemoveContainer" containerID="990016db7bc2477018f6cfda052d6e1f2ef80cd8c99ea3b2051bb70256e5244e"
Nov 03 23:45:24.183336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.183313    5074 scope.go:122] "RemoveContainer" containerID="556b7c6f3058bcfd99e5287c3b72ffdc1a98db8b4af1b2bdf00eedb30afddc86"
Nov 03 23:45:24.188551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.188526    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"]
Nov 03 23:45:24.214126 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.214095    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"]
Nov 03 23:45:24.227524 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.227494    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895825-d8xvc"]
Nov 03 23:45:24.351319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.351287    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:24.446759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.434384    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv\" (UniqueName: \"kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv\") pod \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") "
Nov 03 23:45:24.446759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.434598    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca\") pod \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\" (UID: \"3e30a80b-6eaf-4b13-abda-130da48dbf3e\") "
Nov 03 23:45:24.446759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.435661    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca" pod "3e30a80b-6eaf-4b13-abda-130da48dbf3e" (UID: "3e30a80b-6eaf-4b13-abda-130da48dbf3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:24.446759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.438192    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv" pod "3e30a80b-6eaf-4b13-abda-130da48dbf3e" (UID: "3e30a80b-6eaf-4b13-abda-130da48dbf3e"). InnerVolumeSpecName "kube-api-access-zbpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:24.543810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.539819    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbpqv\" (UniqueName: \"kubernetes.io/projected/3e30a80b-6eaf-4b13-abda-130da48dbf3e-kube-api-access-zbpqv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:24.544045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:24.544022    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30a80b-6eaf-4b13-abda-130da48dbf3e-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:25.021521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.021479    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" path="/var/lib/kubelet/pods/2ce657cd-200f-494a-b6dc-9005e58e93ae/volumes"
Nov 03 23:45:25.164374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.157778    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"
Nov 03 23:45:25.164374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.158521    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2" event={"ID":"3e30a80b-6eaf-4b13-abda-130da48dbf3e","Type":"ContainerDied","Data":"a081b45235fe424671f44be81a2f5315d3938bfc0dabf4e8a8ac936fd1b35af5"}
Nov 03 23:45:25.164374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.158589    5074 scope.go:122] "RemoveContainer" containerID="d162c6b7a7e5dd1edc78bbe0eecf16d1362eaa29f1dcb3d003166966090649d0"
Nov 03 23:45:25.196258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.196206    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"]
Nov 03 23:45:25.370717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.356783    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"]
Nov 03 23:45:25.373969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.373913    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895825-cnbt2"]
Nov 03 23:45:25.648806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.648306    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:25.854762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.853826    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca\") pod \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") "
Nov 03 23:45:25.854762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.853889    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq\" (UniqueName: \"kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq\") pod \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\" (UID: \"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b\") "
Nov 03 23:45:25.922808 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:25.916096    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca" pod "5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" (UID: "5c14555e-0ccb-4ee5-8e38-76fa553b6b1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:26.056765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.053991    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq" pod "5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" (UID: "5c14555e-0ccb-4ee5-8e38-76fa553b6b1b"). InnerVolumeSpecName "kube-api-access-mhlzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:26.379567 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.379529    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:26.449007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.448974    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:26.526951 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.500902    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlzq\" (UniqueName: \"kubernetes.io/projected/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b-kube-api-access-mhlzq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:26.601642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.601592    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls" event={"ID":"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6","Type":"ContainerDied","Data":"710765f0af42f7164ca5f5639309b3aec9c1f7de81878bd42cc48a8f2e0b1fa4"}
Nov 03 23:45:26.626877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.626841    5074 scope.go:122] "RemoveContainer" containerID="a5f2e56d67687dd53abde576d214e90ea24eef2336035936cb854cb154e01e68"
Nov 03 23:45:26.627300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.606379    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"
Nov 03 23:45:26.655813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.607189    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca\") pod \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") "
Nov 03 23:45:26.656420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.656384    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l\" (UniqueName: \"kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l\") pod \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\" (UID: \"9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6\") "
Nov 03 23:45:26.706166 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.706119    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 23:45:26.719969 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.719931    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca" pod "9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" (UID: "9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:26.789830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.778700    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l" pod "9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" (UID: "9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6"). InnerVolumeSpecName "kube-api-access-sv42l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:26.789830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.787778    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv42l\" (UniqueName: \"kubernetes.io/projected/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-kube-api-access-sv42l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:26.789830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.787815    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:26.789830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.789375    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg" event={"ID":"5c14555e-0ccb-4ee5-8e38-76fa553b6b1b","Type":"ContainerDied","Data":"803640adff49cfe9f7c53d1ba4566216616889aa36d39cc9d8cd5ce9e45ca8a5"}
Nov 03 23:45:26.789830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.789794    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"
Nov 03 23:45:26.793859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:26.793824    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.198765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.195256    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" path="/var/lib/kubelet/pods/3e30a80b-6eaf-4b13-abda-130da48dbf3e/volumes"
Nov 03 23:45:27.198765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.196073    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"]
Nov 03 23:45:27.198765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.198674    5074 scope.go:122] "RemoveContainer" containerID="44d5bda9f55daa9a8fee23ae298d581b223799beb0483983b25dc21f0de0afd4"
Nov 03 23:45:27.235967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.225840    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.259247 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.259208    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895825-mrnhg"]
Nov 03 23:45:27.266810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.262936    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 23:45:27.317765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.294700    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"]
Nov 03 23:45:27.342294 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.342235    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895825-bjwls"]
Nov 03 23:45:27.437765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.436861    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.775769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.772089    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.853165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.823912    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:45:27.853165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.780067    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.853165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.781433    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.004848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.990937    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:28.004848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:27.990997    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.054122 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.046852    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:28.054122 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.046922    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.233655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.233611    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:28.234036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.234012    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.234190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.233942    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.234686 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.234663    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:45:28.284764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.274275    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.284764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.274651    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:45:28.487822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.440795    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.487822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.459254    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:45:28.524003 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:28.523955    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:45:28.610323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.609860    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:28.610323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.609928    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.729618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.729581    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:45:28.928818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.916841    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.928818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.917225    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.928818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.917723    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:45:28.928818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:28.917802    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:45:29.129225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.124842    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.129225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.128312    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" path="/var/lib/kubelet/pods/5c14555e-0ccb-4ee5-8e38-76fa553b6b1b/volumes"
Nov 03 23:45:29.338794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.258175    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.338794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.290510    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" path="/var/lib/kubelet/pods/9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6/volumes"
Nov 03 23:45:29.338794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.291365    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 23:45:29.495153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.495105    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.950919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.950868    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.956987 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:29.955015    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 23:45:30.267005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:30.266903    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.304903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:30.304855    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.586044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:30.585939    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.952908 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:30.952866    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:45:30.985102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:30.982289    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:31.211081 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:31.210995    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:45:31.211586 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:31.211567    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:45:31.212609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:31.212591    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:45:31.587190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:31.581630    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:45:31.590701 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:31.588983    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:45:55.240501 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:55.240464    5074 scope.go:122] "RemoveContainer" containerID="cd8b79781b2d42732bb4dc1dea4fcdd190440aa288fab9c08145bbf5db1f4546"
Nov 03 23:45:55.257123 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:55.257097    5074 scope.go:122] "RemoveContainer" containerID="be75f69877dfc339ba4684e43168eeafeefd647d446f4539535e2883dfebee84"
Nov 03 23:45:55.275480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:55.275454    5074 scope.go:122] "RemoveContainer" containerID="79e1d0962fdb72db321505f87db0abcd03a8e244eaf41cfb81d3bf8c8f9b3e96"
Nov 03 23:45:55.287822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:55.287799    5074 scope.go:122] "RemoveContainer" containerID="e90980d024912e180db8734d718b9bad0ba7fd19e05492f3c5a8eef4760e58b7"
Nov 03 23:45:55.308155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:55.308027    5074 scope.go:122] "RemoveContainer" containerID="5ec0ece8f00b973b9a227b2e0efdb3eec53d5cea1c52d27c00804ed325135a30"
Nov 03 23:45:58.061032 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:45:58.060994    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:45:59.449925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.449881    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"]
Nov 03 23:45:59.450571 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450549    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:45:59.450571 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450570    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450586    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450593    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450603    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450611    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450628    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450634    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450677    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450684    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450696    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450703    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450711    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450718    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450755    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450763    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450774    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450781    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450796    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450807    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450821    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450827    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.450835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450843    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d3f16b6b-5302-48c0-819b-757f3e34ea20" containerName="revoke-keys"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.450849    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f16b6b-5302-48c0-819b-757f3e34ea20" containerName="revoke-keys"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451021    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451032    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451043    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451053    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451062    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f16b6b-5302-48c0-819b-757f3e34ea20" containerName="revoke-keys"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451074    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451083    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451091    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:45:59.451850 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.451099    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:45:59.457532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.457507    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.464262 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.464237    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"]
Nov 03 23:45:59.540658 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.540615    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.540992    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541115 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541085    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541305 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541285    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541328    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541410    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541444    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.541615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.541599    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpdzn\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642339    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpdzn\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642404    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642462    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642494    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642754 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642643    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642754 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642678    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642754 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642704    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642798    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.642919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642832    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.643028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.642951    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.643122 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.643079    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.643238 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.643218    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.643290 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.643269    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.643724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.643695    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.644979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.644947    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.645287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.645266    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.655161 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.655140    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.655276 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.655194    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpdzn\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:45:59.788815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:45:59.788699    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:46:00.098234 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:46:00.098188    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3abbca_9c07_4b0e_a9d0_0544fedffc27.slice/crio-ef80489c36445a03e89d00a95ad7c2d8cc60e22fed327a9535a47f832fc9657f WatchSource:0}: Error finding container ef80489c36445a03e89d00a95ad7c2d8cc60e22fed327a9535a47f832fc9657f: Status 404 returned error can't find the container with id ef80489c36445a03e89d00a95ad7c2d8cc60e22fed327a9535a47f832fc9657f
Nov 03 23:46:00.098471 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.098451    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"]
Nov 03 23:46:00.100873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.100840    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:46:00.100981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.100903    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:46:00.100981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.100934    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:46:00.323939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.323902    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" event={"ID":"2e3abbca-9c07-4b0e-a9d0-0544fedffc27","Type":"ContainerStarted","Data":"f3629049f7bb80e1166a03027682cf80e70500f6e9280f889384b7bd95f24c9e"}
Nov 03 23:46:00.324113 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.323961    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" event={"ID":"2e3abbca-9c07-4b0e-a9d0-0544fedffc27","Type":"ContainerStarted","Data":"ef80489c36445a03e89d00a95ad7c2d8cc60e22fed327a9535a47f832fc9657f"}
Nov 03 23:46:00.350132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.350022    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podStartSLOduration=1.35000688 podStartE2EDuration="1.35000688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:00.348391213 +0000 UTC m=+1025.447991357" watchObservedRunningTime="2026-11-03 23:46:00.35000688 +0000 UTC m=+1025.449607024"
Nov 03 23:46:00.789234 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:00.789203    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:46:01.790985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:01.790953    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 23:46:01.791388 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:01.791004    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 23:46:02.789664 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:02.789633    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 23:46:02.789976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:02.789947    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 23:46:03.789972 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:03.789932    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 23:46:03.790562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:03.790517    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 23:46:04.790985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:04.789906    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 23:46:04.790985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:04.789989    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 23:46:05.789133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:05.789098    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 23:46:05.789349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:05.789325    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 23:46:06.789785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:06.789733    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 23:46:06.790286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:06.789823    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 23:46:07.794044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:07.794016    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:46:07.794540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:07.794337    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:46:07.795153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:07.795134    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:46:14.499031 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.498991    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"]
Nov 03 23:46:14.502164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.499994    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:46:14.502164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.500020    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:46:14.502164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.500231    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c14555e-0ccb-4ee5-8e38-76fa553b6b1b" containerName="cleanup"
Nov 03 23:46:14.502164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.500246    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30a80b-6eaf-4b13-abda-130da48dbf3e" containerName="cleanup"
Nov 03 23:46:14.502164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.500264    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8c8e46-dbbe-4d84-9d05-22d6629cd2c6" containerName="cleanup"
Nov 03 23:46:14.502164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.500276    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce657cd-200f-494a-b6dc-9005e58e93ae" containerName="cleanup"
Nov 03 23:46:14.505378 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.505255    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.514958 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.514931    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"]
Nov 03 23:46:14.560423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560368    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.560669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560442    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.560669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560488    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.560669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560522    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.560669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560550    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.560669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560634    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.560669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560667    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkb2z\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.561253 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560702    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.561253 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.560848    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.662636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.662601    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.662840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.662646    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkb2z\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.662840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.662673    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.662840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.662735    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.662860    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.662901    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663049    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663075    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663099    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663322 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663149    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663322 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663316    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663432    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.663677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.663659    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.665327 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.665304    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.665536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.665513    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.677147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.677118    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.677373 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.677349    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkb2z\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:14.832847 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:14.832764    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:15.165513 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:46:15.165474    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9db13d_1bd6_4f71_8a6a_d512e4e36970.slice/crio-6261c4035eaba27bd07677d6d557e67f82d727cf058d09b65d485d4fb67a9e7e WatchSource:0}: Error finding container 6261c4035eaba27bd07677d6d557e67f82d727cf058d09b65d485d4fb67a9e7e: Status 404 returned error can't find the container with id 6261c4035eaba27bd07677d6d557e67f82d727cf058d09b65d485d4fb67a9e7e
Nov 03 23:46:15.166344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.166321    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"]
Nov 03 23:46:15.168285 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.168252    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:46:15.168358 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.168331    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:46:15.168396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.168362    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:46:15.552210 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.552174    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" event={"ID":"5b9db13d-1bd6-4f71-8a6a-d512e4e36970","Type":"ContainerStarted","Data":"67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f"}
Nov 03 23:46:15.552701 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.552242    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" event={"ID":"5b9db13d-1bd6-4f71-8a6a-d512e4e36970","Type":"ContainerStarted","Data":"6261c4035eaba27bd07677d6d557e67f82d727cf058d09b65d485d4fb67a9e7e"}
Nov 03 23:46:15.605767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.586268    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podStartSLOduration=1.586242057 podStartE2EDuration="1.586242057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:15.580053265 +0000 UTC m=+1040.679653410" watchObservedRunningTime="2026-11-03 23:46:15.586242057 +0000 UTC m=+1040.685842206"
Nov 03 23:46:15.833778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:15.833666    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:16.836761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:16.836565    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:46:16.836761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:16.836649    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 23:46:17.834083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:17.834037    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:46:17.834296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:17.834110    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 23:46:18.841766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:18.839142    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:46:18.841766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:18.839205    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 23:46:19.833680 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:19.833260    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:46:19.833680 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:19.833320    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 23:46:20.834730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:20.834233    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:46:20.834730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:20.834313    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 23:46:21.844555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:21.844496    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:46:21.845464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:21.845412    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 23:46:23.160813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:23.154124    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:23.160813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:23.154546    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:23.160813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:23.156633    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 03 23:46:28.024521 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:46:28.024482    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:46:31.637169 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.637135    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"]
Nov 03 23:46:31.643043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.643022    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.650413 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.650341    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:46:31.651351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.651324    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"]
Nov 03 23:46:31.652422 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.652380    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:46:31.703145 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.703110    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.703292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.703266    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c6m\" (UniqueName: \"kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.703399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.703382    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.805467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.805428    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.805637 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.805555    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c6m\" (UniqueName: \"kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.805685 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.805639    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.806124 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.806096    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.808141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.808116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.819636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.819608    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c6m\" (UniqueName: \"kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m\") pod \"maas-api-revoke-sub-875d96be-tsv8r\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:31.972147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:31.972117    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:32.373892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:32.373858    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"]
Nov 03 23:46:32.376289 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:46:32.376260    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7059414f_1b8f_4233_9cff_dbf154466754.slice/crio-6b07e5659b99d805992fd45d49ce30edc2438823463082174f3dd7e68a52e4e8 WatchSource:0}: Error finding container 6b07e5659b99d805992fd45d49ce30edc2438823463082174f3dd7e68a52e4e8: Status 404 returned error can't find the container with id 6b07e5659b99d805992fd45d49ce30edc2438823463082174f3dd7e68a52e4e8
Nov 03 23:46:32.765326 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:32.765292    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r" event={"ID":"7059414f-1b8f-4233-9cff-dbf154466754","Type":"ContainerStarted","Data":"e967ac97cd7ddb6123321f799900623231ab1cab17e9f5930b589489a7088d2d"}
Nov 03 23:46:32.765838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:32.765347    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r" event={"ID":"7059414f-1b8f-4233-9cff-dbf154466754","Type":"ContainerStarted","Data":"6b07e5659b99d805992fd45d49ce30edc2438823463082174f3dd7e68a52e4e8"}
Nov 03 23:46:32.798903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:32.798843    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r" podStartSLOduration=1.798824717 podStartE2EDuration="1.798824717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:32.793545145 +0000 UTC m=+1057.893145290" watchObservedRunningTime="2026-11-03 23:46:32.798824717 +0000 UTC m=+1057.898424864"
Nov 03 23:46:33.046578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:33.046462    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"]
Nov 03 23:46:33.051466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:33.051433    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c0bc15-ng7lh"]
Nov 03 23:46:33.774892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:33.774859    5074 generic.go:356] "Generic (PLEG): container finished" podID="7059414f-1b8f-4233-9cff-dbf154466754" containerID="e967ac97cd7ddb6123321f799900623231ab1cab17e9f5930b589489a7088d2d" exitCode=0
Nov 03 23:46:33.775325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:33.774906    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r" event={"ID":"7059414f-1b8f-4233-9cff-dbf154466754","Type":"ContainerDied","Data":"e967ac97cd7ddb6123321f799900623231ab1cab17e9f5930b589489a7088d2d"}
Nov 03 23:46:35.335789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.328267    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d" path="/var/lib/kubelet/pods/0dd4964b-7ec4-4a92-a907-e6f3d1fddb9d/volumes"
Nov 03 23:46:35.350898 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.350866    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:35.386363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.386326    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca\") pod \"7059414f-1b8f-4233-9cff-dbf154466754\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") "
Nov 03 23:46:35.386657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.386638    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m\" (UniqueName: \"kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m\") pod \"7059414f-1b8f-4233-9cff-dbf154466754\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") "
Nov 03 23:46:35.386813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.386797    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca\") pod \"7059414f-1b8f-4233-9cff-dbf154466754\" (UID: \"7059414f-1b8f-4233-9cff-dbf154466754\") "
Nov 03 23:46:35.387829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.387804    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca" pod "7059414f-1b8f-4233-9cff-dbf154466754" (UID: "7059414f-1b8f-4233-9cff-dbf154466754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:35.395790 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.390731    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca" pod "7059414f-1b8f-4233-9cff-dbf154466754" (UID: "7059414f-1b8f-4233-9cff-dbf154466754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:46:35.420885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.419254    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m" pod "7059414f-1b8f-4233-9cff-dbf154466754" (UID: "7059414f-1b8f-4233-9cff-dbf154466754"). InnerVolumeSpecName "kube-api-access-m7c6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:35.488581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.488543    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:35.488581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.488582    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c6m\" (UniqueName: \"kubernetes.io/projected/7059414f-1b8f-4233-9cff-dbf154466754-kube-api-access-m7c6m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:35.488793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.488602    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7059414f-1b8f-4233-9cff-dbf154466754-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:35.795681 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.795650    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"
Nov 03 23:46:35.795898 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.795679    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r" event={"ID":"7059414f-1b8f-4233-9cff-dbf154466754","Type":"ContainerDied","Data":"6b07e5659b99d805992fd45d49ce30edc2438823463082174f3dd7e68a52e4e8"}
Nov 03 23:46:35.795898 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:35.795751    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b07e5659b99d805992fd45d49ce30edc2438823463082174f3dd7e68a52e4e8"
Nov 03 23:46:42.028653 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:42.028615    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"]
Nov 03 23:46:42.033375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:42.033348    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1795fca2-8t84l"]
Nov 03 23:46:43.019156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:43.019116    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32080b3-f130-47b7-ab1c-f3a20504de7c" path="/var/lib/kubelet/pods/c32080b3-f130-47b7-ab1c-f3a20504de7c/volumes"
Nov 03 23:46:43.030392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:43.030352    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"]
Nov 03 23:46:43.034826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:43.034804    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb01ec-g9g9v"]
Nov 03 23:46:45.019089 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:45.019055    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110009d6-c3ce-4f74-9c0b-fcbddfc390f4" path="/var/lib/kubelet/pods/110009d6-c3ce-4f74-9c0b-fcbddfc390f4/volumes"
Nov 03 23:46:46.038375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:46.038339    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"]
Nov 03 23:46:46.042546 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:46.042514    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881dcbc4-nhhwc"]
Nov 03 23:46:47.029408 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:47.029358    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177ba727-56bb-496a-82a7-4bafe41aab8a" path="/var/lib/kubelet/pods/177ba727-56bb-496a-82a7-4bafe41aab8a/volumes"
Nov 03 23:46:54.028994 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:54.028961    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"]
Nov 03 23:46:54.033373 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:54.033344    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62b574d4-hhh4j"]
Nov 03 23:46:55.019553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:55.019522    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0910b88f-490a-4321-ac53-144426420ead" path="/var/lib/kubelet/pods/0910b88f-490a-4321-ac53-144426420ead/volumes"
Nov 03 23:46:56.102399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:56.102366    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"]
Nov 03 23:46:56.109070 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:56.109041    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827fbe14-zvvjv"]
Nov 03 23:46:57.022999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:46:57.022958    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0036ea50-7f0a-4f60-8bac-e8af2b7235b0" path="/var/lib/kubelet/pods/0036ea50-7f0a-4f60-8bac-e8af2b7235b0/volumes"
Nov 03 23:46:58.149768 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:46:58.144333    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:47:06.035938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:06.035903    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"]
Nov 03 23:47:06.039918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:06.039892    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192de9-jbbj9"]
Nov 03 23:47:07.019832 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:07.019797    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0185379-8e2f-4f3c-b9dc-bae20ac7f718" path="/var/lib/kubelet/pods/d0185379-8e2f-4f3c-b9dc-bae20ac7f718/volumes"
Nov 03 23:47:07.030986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:07.030949    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"]
Nov 03 23:47:07.037346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:07.037319    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d203e-7h9t4"]
Nov 03 23:47:09.020327 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:09.020286    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168723c6-54e6-426b-ac7e-4e596efd2a31" path="/var/lib/kubelet/pods/168723c6-54e6-426b-ac7e-4e596efd2a31/volumes"
Nov 03 23:47:11.963008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.962973    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"]
Nov 03 23:47:11.963578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.963564    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7059414f-1b8f-4233-9cff-dbf154466754" containerName="revoke-keys"
Nov 03 23:47:11.963622 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.963578    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7059414f-1b8f-4233-9cff-dbf154466754" containerName="revoke-keys"
Nov 03 23:47:11.963767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.963726    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7059414f-1b8f-4233-9cff-dbf154466754" containerName="revoke-keys"
Nov 03 23:47:11.967234 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.967212    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:11.974992 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.974966    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:47:11.976638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.976616    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:47:11.979793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:11.979770    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"]
Nov 03 23:47:12.048357 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.048325    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7h7\" (UniqueName: \"kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.048549 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.048517    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.048612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.048556    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.150135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.150092    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.150344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.150164    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.150407 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.150343    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7h7\" (UniqueName: \"kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.151069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.151044    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.152945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.152910    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.163416 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.163388    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7h7\" (UniqueName: \"kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7\") pod \"maas-api-revoke-sub-d9cb4681-k9nnb\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.283473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.283374    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:12.620935 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:12.620899    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"]
Nov 03 23:47:12.621824 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:47:12.621795    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d16fe35_ca87_45d9_aad8_be4149b6b2bd.slice/crio-7136208f4249a448f0356ec11804d6d970ec7efffbb34b82bf8f75604de6959a WatchSource:0}: Error finding container 7136208f4249a448f0356ec11804d6d970ec7efffbb34b82bf8f75604de6959a: Status 404 returned error can't find the container with id 7136208f4249a448f0356ec11804d6d970ec7efffbb34b82bf8f75604de6959a
Nov 03 23:47:13.401587 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:13.401549    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb" event={"ID":"1d16fe35-ca87-45d9-aad8-be4149b6b2bd","Type":"ContainerStarted","Data":"04c37792efbafb8c7648cc1e09b62c7f7b91eca19689fe2e279a60e24a993b30"}
Nov 03 23:47:13.402115 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:13.401615    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb" event={"ID":"1d16fe35-ca87-45d9-aad8-be4149b6b2bd","Type":"ContainerStarted","Data":"7136208f4249a448f0356ec11804d6d970ec7efffbb34b82bf8f75604de6959a"}
Nov 03 23:47:13.424770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:13.424692    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb" podStartSLOduration=2.424676412 podStartE2EDuration="2.424676412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:13.423384699 +0000 UTC m=+1098.522984843" watchObservedRunningTime="2026-11-03 23:47:13.424676412 +0000 UTC m=+1098.524276579"
Nov 03 23:47:14.411954 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:14.411911    5074 generic.go:356] "Generic (PLEG): container finished" podID="1d16fe35-ca87-45d9-aad8-be4149b6b2bd" containerID="04c37792efbafb8c7648cc1e09b62c7f7b91eca19689fe2e279a60e24a993b30" exitCode=0
Nov 03 23:47:14.412350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:14.412001    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb" event={"ID":"1d16fe35-ca87-45d9-aad8-be4149b6b2bd","Type":"ContainerDied","Data":"04c37792efbafb8c7648cc1e09b62c7f7b91eca19689fe2e279a60e24a993b30"}
Nov 03 23:47:15.589193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.589165    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:15.706650 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.706607    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca\") pod \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") "
Nov 03 23:47:15.706855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.706843    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7\" (UniqueName: \"kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7\") pod \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") "
Nov 03 23:47:15.706988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.706956    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca\") pod \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\" (UID: \"1d16fe35-ca87-45d9-aad8-be4149b6b2bd\") "
Nov 03 23:47:15.707363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.707334    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca" pod "1d16fe35-ca87-45d9-aad8-be4149b6b2bd" (UID: "1d16fe35-ca87-45d9-aad8-be4149b6b2bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:15.707600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.707572    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:15.709048 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.709021    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca" pod "1d16fe35-ca87-45d9-aad8-be4149b6b2bd" (UID: "1d16fe35-ca87-45d9-aad8-be4149b6b2bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:47:15.709175 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.709152    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7" pod "1d16fe35-ca87-45d9-aad8-be4149b6b2bd" (UID: "1d16fe35-ca87-45d9-aad8-be4149b6b2bd"). InnerVolumeSpecName "kube-api-access-rq7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:15.808713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.808676    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:15.808713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:15.808712    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq7h7\" (UniqueName: \"kubernetes.io/projected/1d16fe35-ca87-45d9-aad8-be4149b6b2bd-kube-api-access-rq7h7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:16.430205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:16.430172    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"
Nov 03 23:47:16.430410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:16.430213    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb" event={"ID":"1d16fe35-ca87-45d9-aad8-be4149b6b2bd","Type":"ContainerDied","Data":"7136208f4249a448f0356ec11804d6d970ec7efffbb34b82bf8f75604de6959a"}
Nov 03 23:47:16.430410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:16.430303    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7136208f4249a448f0356ec11804d6d970ec7efffbb34b82bf8f75604de6959a"
Nov 03 23:47:17.437844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.437806    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"]
Nov 03 23:47:17.438539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.438521    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1d16fe35-ca87-45d9-aad8-be4149b6b2bd" containerName="revoke-keys"
Nov 03 23:47:17.438612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.438541    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d16fe35-ca87-45d9-aad8-be4149b6b2bd" containerName="revoke-keys"
Nov 03 23:47:17.438682 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.438673    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d16fe35-ca87-45d9-aad8-be4149b6b2bd" containerName="revoke-keys"
Nov 03 23:47:17.442771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.442725    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.450085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.450057    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:47:17.450249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.450092    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:47:17.510576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.510538    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"]
Nov 03 23:47:17.533084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.533043    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.533327 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.533304    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.533469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.533451    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvmd\" (UniqueName: \"kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.634777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.634721    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.635150 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.635122    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.635258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.635241    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvmd\" (UniqueName: \"kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.635937 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.635911    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.637927 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.637896    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.654008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.653972    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvmd\" (UniqueName: \"kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd\") pod \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:17.759107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:17.759025    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:18.051675 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:47:18.051640    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497374cb_b862_467a_bc3c_e1a6f011a23e.slice/crio-848cde52e0d3aad9408bfed9e812f4aef85329c8d1a1837bdeedbf217aae92c5 WatchSource:0}: Error finding container 848cde52e0d3aad9408bfed9e812f4aef85329c8d1a1837bdeedbf217aae92c5: Status 404 returned error can't find the container with id 848cde52e0d3aad9408bfed9e812f4aef85329c8d1a1837bdeedbf217aae92c5
Nov 03 23:47:18.051823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:18.051675    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"]
Nov 03 23:47:18.451261 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:18.451215    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4" event={"ID":"497374cb-b862-467a-bc3c-e1a6f011a23e","Type":"ContainerStarted","Data":"d6346fb09ce7464b553bac60b6313aacb159159449d15c350f4d6cd959a927cf"}
Nov 03 23:47:18.451767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:18.451273    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4" event={"ID":"497374cb-b862-467a-bc3c-e1a6f011a23e","Type":"ContainerStarted","Data":"848cde52e0d3aad9408bfed9e812f4aef85329c8d1a1837bdeedbf217aae92c5"}
Nov 03 23:47:18.476282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:18.476221    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4" podStartSLOduration=1.4762081120000001 podStartE2EDuration="1.476208112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:18.473610242 +0000 UTC m=+1103.573210386" watchObservedRunningTime="2026-11-03 23:47:18.476208112 +0000 UTC m=+1103.575808261"
Nov 03 23:47:19.460087 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:19.460049    5074 generic.go:356] "Generic (PLEG): container finished" podID="497374cb-b862-467a-bc3c-e1a6f011a23e" containerID="d6346fb09ce7464b553bac60b6313aacb159159449d15c350f4d6cd959a927cf" exitCode=0
Nov 03 23:47:19.460494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:19.460130    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4" event={"ID":"497374cb-b862-467a-bc3c-e1a6f011a23e","Type":"ContainerDied","Data":"d6346fb09ce7464b553bac60b6313aacb159159449d15c350f4d6cd959a927cf"}
Nov 03 23:47:20.644290 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.644261    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:20.796797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.796704    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca\") pod \"497374cb-b862-467a-bc3c-e1a6f011a23e\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") "
Nov 03 23:47:20.796797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.796781    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca\") pod \"497374cb-b862-467a-bc3c-e1a6f011a23e\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") "
Nov 03 23:47:20.796980 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.796834    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd\" (UniqueName: \"kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd\") pod \"497374cb-b862-467a-bc3c-e1a6f011a23e\" (UID: \"497374cb-b862-467a-bc3c-e1a6f011a23e\") "
Nov 03 23:47:20.797209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.797165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca" pod "497374cb-b862-467a-bc3c-e1a6f011a23e" (UID: "497374cb-b862-467a-bc3c-e1a6f011a23e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:20.797415 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.797279    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:20.798765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.798728    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca" pod "497374cb-b862-467a-bc3c-e1a6f011a23e" (UID: "497374cb-b862-467a-bc3c-e1a6f011a23e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:47:20.798860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.798851    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd" pod "497374cb-b862-467a-bc3c-e1a6f011a23e" (UID: "497374cb-b862-467a-bc3c-e1a6f011a23e"). InnerVolumeSpecName "kube-api-access-cgvmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:20.898503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.898466    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgvmd\" (UniqueName: \"kubernetes.io/projected/497374cb-b862-467a-bc3c-e1a6f011a23e-kube-api-access-cgvmd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:20.898503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:20.898498    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497374cb-b862-467a-bc3c-e1a6f011a23e-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:21.479432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:21.479400    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:21.479624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:21.479443    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4" event={"ID":"497374cb-b862-467a-bc3c-e1a6f011a23e","Type":"ContainerDied","Data":"848cde52e0d3aad9408bfed9e812f4aef85329c8d1a1837bdeedbf217aae92c5"}
Nov 03 23:47:21.479624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:21.479503    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848cde52e0d3aad9408bfed9e812f4aef85329c8d1a1837bdeedbf217aae92c5"
Nov 03 23:47:28.027122 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:47:28.027085    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:47:31.039309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:31.039271    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"]
Nov 03 23:47:31.043504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:31.043472    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a34131-cz9js"]
Nov 03 23:47:33.021905 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.021870    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146fdda6-d0c2-4267-8c5f-cd313132060d" path="/var/lib/kubelet/pods/146fdda6-d0c2-4267-8c5f-cd313132060d/volumes"
Nov 03 23:47:33.332622 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.332524    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"]
Nov 03 23:47:33.333621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.333594    5074 cpu_manager.go:414] "containerMap: removing container" podUID="497374cb-b862-467a-bc3c-e1a6f011a23e" containerName="revoke-keys"
Nov 03 23:47:33.333787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.333621    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497374cb-b862-467a-bc3c-e1a6f011a23e" containerName="revoke-keys"
Nov 03 23:47:33.333851 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.333811    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="497374cb-b862-467a-bc3c-e1a6f011a23e" containerName="revoke-keys"
Nov 03 23:47:33.339862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.339839    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.344405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.344380    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"]
Nov 03 23:47:33.351961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.351937    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:47:33.352090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.351989    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:47:33.449287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.449253    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f68gw\" (UniqueName: \"kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.449468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.449317    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.449535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.449514    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.550706 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.550675    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.550885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.550805    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f68gw\" (UniqueName: \"kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.550885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.550852    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.551468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.551449    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.553337 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.553318    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.573831 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.573808    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f68gw\" (UniqueName: \"kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw\") pod \"maas-api-revoke-sub-824ba939-zsvp2\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.658160 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.657564    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:33.977473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:33.977445    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"]
Nov 03 23:47:33.979785 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:47:33.979761    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2bada4_b700_4183_af93_093d8a6ed0b4.slice/crio-fdb582d390785285360def18c91601f420f492ecda5c7bfad2896ef7a82058e8 WatchSource:0}: Error finding container fdb582d390785285360def18c91601f420f492ecda5c7bfad2896ef7a82058e8: Status 404 returned error can't find the container with id fdb582d390785285360def18c91601f420f492ecda5c7bfad2896ef7a82058e8
Nov 03 23:47:34.829596 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:34.829504    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2" event={"ID":"3b2bada4-b700-4183-af93-093d8a6ed0b4","Type":"ContainerStarted","Data":"1c1a5973177363609537bdf1cb7e09e0a8d6ffb9870bc521ed5c28a41ba7bba1"}
Nov 03 23:47:34.829596 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:34.829577    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2" event={"ID":"3b2bada4-b700-4183-af93-093d8a6ed0b4","Type":"ContainerStarted","Data":"fdb582d390785285360def18c91601f420f492ecda5c7bfad2896ef7a82058e8"}
Nov 03 23:47:34.853274 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:34.853220    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2" podStartSLOduration=1.85320525 podStartE2EDuration="1.85320525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:34.84866255 +0000 UTC m=+1119.948262694" watchObservedRunningTime="2026-11-03 23:47:34.85320525 +0000 UTC m=+1119.952805444"
Nov 03 23:47:35.753859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.753818    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"]
Nov 03 23:47:35.758165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.758146    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.766032 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.766005    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"]
Nov 03 23:47:35.840687 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.840654    5074 generic.go:356] "Generic (PLEG): container finished" podID="3b2bada4-b700-4183-af93-093d8a6ed0b4" containerID="1c1a5973177363609537bdf1cb7e09e0a8d6ffb9870bc521ed5c28a41ba7bba1" exitCode=0
Nov 03 23:47:35.841065 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.840756    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2" event={"ID":"3b2bada4-b700-4183-af93-093d8a6ed0b4","Type":"ContainerDied","Data":"1c1a5973177363609537bdf1cb7e09e0a8d6ffb9870bc521ed5c28a41ba7bba1"}
Nov 03 23:47:35.889867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.889833    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.890029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.889982    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.890203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.890187    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd52c\" (UniqueName: \"kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.991159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.991124    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd52c\" (UniqueName: \"kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.991325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.991174    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.991325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.991283    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.992019 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.991992    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:35.993892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:35.993856    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:36.005582 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:36.005528    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd52c\" (UniqueName: \"kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c\") pod \"maas-api-revoke-sub-89f338e4-5mmq7\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:36.077636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:36.077604    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:36.365663 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:47:36.365631    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d7eee2_545e_475f_9ea5_8a705b1aae06.slice/crio-c0856cb353c3e78d247a7aa67d00667d845b4018614501aa45585ea1c7f3acb8 WatchSource:0}: Error finding container c0856cb353c3e78d247a7aa67d00667d845b4018614501aa45585ea1c7f3acb8: Status 404 returned error can't find the container with id c0856cb353c3e78d247a7aa67d00667d845b4018614501aa45585ea1c7f3acb8
Nov 03 23:47:36.366399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:36.366366    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"]
Nov 03 23:47:36.861825 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:36.861781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7" event={"ID":"98d7eee2-545e-475f-9ea5-8a705b1aae06","Type":"ContainerStarted","Data":"c0856cb353c3e78d247a7aa67d00667d845b4018614501aa45585ea1c7f3acb8"}
Nov 03 23:47:37.198151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.198126    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:37.244084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.244051    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca\") pod \"3b2bada4-b700-4183-af93-093d8a6ed0b4\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") "
Nov 03 23:47:37.244512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.244486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca\") pod \"3b2bada4-b700-4183-af93-093d8a6ed0b4\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") "
Nov 03 23:47:37.244688 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.244669    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw\" (UniqueName: \"kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw\") pod \"3b2bada4-b700-4183-af93-093d8a6ed0b4\" (UID: \"3b2bada4-b700-4183-af93-093d8a6ed0b4\") "
Nov 03 23:47:37.244872 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.244842    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca" pod "3b2bada4-b700-4183-af93-093d8a6ed0b4" (UID: "3b2bada4-b700-4183-af93-093d8a6ed0b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:37.245473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.245446    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:37.248463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.247908    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca" pod "3b2bada4-b700-4183-af93-093d8a6ed0b4" (UID: "3b2bada4-b700-4183-af93-093d8a6ed0b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:47:37.248679 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.248653    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw" pod "3b2bada4-b700-4183-af93-093d8a6ed0b4" (UID: "3b2bada4-b700-4183-af93-093d8a6ed0b4"). InnerVolumeSpecName "kube-api-access-f68gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:37.347394 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.347307    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f68gw\" (UniqueName: \"kubernetes.io/projected/3b2bada4-b700-4183-af93-093d8a6ed0b4-kube-api-access-f68gw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:37.347394 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.347350    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2bada4-b700-4183-af93-093d8a6ed0b4-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:37.870927 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.870892    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"
Nov 03 23:47:37.871454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.870936    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2" event={"ID":"3b2bada4-b700-4183-af93-093d8a6ed0b4","Type":"ContainerDied","Data":"fdb582d390785285360def18c91601f420f492ecda5c7bfad2896ef7a82058e8"}
Nov 03 23:47:37.871454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.871027    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdb582d390785285360def18c91601f420f492ecda5c7bfad2896ef7a82058e8"
Nov 03 23:47:37.873212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.873183    5074 generic.go:356] "Generic (PLEG): container finished" podID="98d7eee2-545e-475f-9ea5-8a705b1aae06" containerID="d45f25336df05b94f2407f512efb469777812b267e21e6327c52f74e03e69e02" exitCode=0
Nov 03 23:47:37.873358 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:37.873277    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7" event={"ID":"98d7eee2-545e-475f-9ea5-8a705b1aae06","Type":"ContainerDied","Data":"d45f25336df05b94f2407f512efb469777812b267e21e6327c52f74e03e69e02"}
Nov 03 23:47:39.058964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.058935    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:39.079361 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.079330    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca\") pod \"98d7eee2-545e-475f-9ea5-8a705b1aae06\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") "
Nov 03 23:47:39.079361 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.079364    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca\") pod \"98d7eee2-545e-475f-9ea5-8a705b1aae06\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") "
Nov 03 23:47:39.079612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.079511    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c\" (UniqueName: \"kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c\") pod \"98d7eee2-545e-475f-9ea5-8a705b1aae06\" (UID: \"98d7eee2-545e-475f-9ea5-8a705b1aae06\") "
Nov 03 23:47:39.079793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.079766    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca" pod "98d7eee2-545e-475f-9ea5-8a705b1aae06" (UID: "98d7eee2-545e-475f-9ea5-8a705b1aae06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:39.080220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.080192    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:39.081659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.081633    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca" pod "98d7eee2-545e-475f-9ea5-8a705b1aae06" (UID: "98d7eee2-545e-475f-9ea5-8a705b1aae06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:47:39.081659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.081647    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c" pod "98d7eee2-545e-475f-9ea5-8a705b1aae06" (UID: "98d7eee2-545e-475f-9ea5-8a705b1aae06"). InnerVolumeSpecName "kube-api-access-kd52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:39.181766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.181641    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d7eee2-545e-475f-9ea5-8a705b1aae06-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:39.181766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.181673    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd52c\" (UniqueName: \"kubernetes.io/projected/98d7eee2-545e-475f-9ea5-8a705b1aae06-kube-api-access-kd52c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:39.890470 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.890439    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"
Nov 03 23:47:39.890620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.890469    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7" event={"ID":"98d7eee2-545e-475f-9ea5-8a705b1aae06","Type":"ContainerDied","Data":"c0856cb353c3e78d247a7aa67d00667d845b4018614501aa45585ea1c7f3acb8"}
Nov 03 23:47:39.890620 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:39.890518    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0856cb353c3e78d247a7aa67d00667d845b4018614501aa45585ea1c7f3acb8"
Nov 03 23:47:45.013763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:45.008302    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"]
Nov 03 23:47:45.024055 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:47:45.024001    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="497374cb-b862-467a-bc3c-e1a6f011a23e" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"
Nov 03 23:47:45.043336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:45.043302    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88f34dbe-fqzg4"]
Nov 03 23:47:47.108507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:47.108465    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497374cb-b862-467a-bc3c-e1a6f011a23e" path="/var/lib/kubelet/pods/497374cb-b862-467a-bc3c-e1a6f011a23e/volumes"
Nov 03 23:47:52.599883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.599845    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"]
Nov 03 23:47:52.600688 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.600666    5074 cpu_manager.go:414] "containerMap: removing container" podUID="98d7eee2-545e-475f-9ea5-8a705b1aae06" containerName="revoke-keys"
Nov 03 23:47:52.600688 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.600685    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d7eee2-545e-475f-9ea5-8a705b1aae06" containerName="revoke-keys"
Nov 03 23:47:52.600924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.600696    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3b2bada4-b700-4183-af93-093d8a6ed0b4" containerName="revoke-keys"
Nov 03 23:47:52.600924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.600702    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2bada4-b700-4183-af93-093d8a6ed0b4" containerName="revoke-keys"
Nov 03 23:47:52.600924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.600837    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d7eee2-545e-475f-9ea5-8a705b1aae06" containerName="revoke-keys"
Nov 03 23:47:52.600924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.600847    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2bada4-b700-4183-af93-093d8a6ed0b4" containerName="revoke-keys"
Nov 03 23:47:52.606334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.606262    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.659975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.659918    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"]
Nov 03 23:47:52.663970 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.663947    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d41841-dockercfg-4d8k7\""
Nov 03 23:47:52.667552 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.667528    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d41841\""
Nov 03 23:47:52.680426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.680394    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5lx\" (UniqueName: \"kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx\") pod \"payload-processing-e2e-api-a-d41841-744c8b765d-7csvr\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") " pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.680692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.680671    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d41841-744c8b765d-7csvr\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") " pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.781775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.781668    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5lx\" (UniqueName: \"kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx\") pod \"payload-processing-e2e-api-a-d41841-744c8b765d-7csvr\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") " pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.781775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.781719    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d41841-744c8b765d-7csvr\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") " pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.784770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.782759    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d41841-744c8b765d-7csvr\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") " pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.804110 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.804084    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5lx\" (UniqueName: \"kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx\") pod \"payload-processing-e2e-api-a-d41841-744c8b765d-7csvr\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") " pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:52.948396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:52.948068    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:53.260014 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:47:53.259988    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad039f7_d961_4068_bc75_d34b87a999a9.slice/crio-ee25b089bc4ee30db487dad1d588815f7011ab652e9da3f116d4d4fa4252d354 WatchSource:0}: Error finding container ee25b089bc4ee30db487dad1d588815f7011ab652e9da3f116d4d4fa4252d354: Status 404 returned error can't find the container with id ee25b089bc4ee30db487dad1d588815f7011ab652e9da3f116d4d4fa4252d354
Nov 03 23:47:53.260634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:53.260606    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"]
Nov 03 23:47:54.078159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:54.077683    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr" event={"ID":"4ad039f7-d961-4068-bc75-d34b87a999a9","Type":"ContainerStarted","Data":"ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578"}
Nov 03 23:47:54.078159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:54.077938    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr" event={"ID":"4ad039f7-d961-4068-bc75-d34b87a999a9","Type":"ContainerStarted","Data":"ee25b089bc4ee30db487dad1d588815f7011ab652e9da3f116d4d4fa4252d354"}
Nov 03 23:47:54.078159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:54.078072    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:47:54.102476 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:54.102427    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr" podStartSLOduration=2.102415173 podStartE2EDuration="2.102415173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:54.098106877 +0000 UTC m=+1139.197707020" watchObservedRunningTime="2026-11-03 23:47:54.102415173 +0000 UTC m=+1139.202015316"
Nov 03 23:47:55.423080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:55.423050    5074 scope.go:122] "RemoveContainer" containerID="2915856aec18487baaef69742df281f5d42fe238e0121748c5308499487185bd"
Nov 03 23:47:55.434233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:55.434212    5074 scope.go:122] "RemoveContainer" containerID="7ef55a1453fc6c3a43ac312044e23e9a065b39ed069e3544bf666e1ad1a917b0"
Nov 03 23:47:55.445894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:55.445871    5074 scope.go:122] "RemoveContainer" containerID="706a63656cf93f111e7b36766b3b4ffc6b5f3fe0852edba44108d8eec49b2747"
Nov 03 23:47:55.455587 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:55.455563    5074 scope.go:122] "RemoveContainer" containerID="4d7159e03f549c19cf1158a2669737393e8648ed4f07d2028be386045d2a1109"
Nov 03 23:47:55.464833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:55.464814    5074 scope.go:122] "RemoveContainer" containerID="651db41aa6bb54c953e53cebce5a68a413bd89ea6052a0ea49456c9d41132f20"
Nov 03 23:47:55.473610 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:47:55.473589    5074 scope.go:122] "RemoveContainer" containerID="b12e6da84501f9b8558afb85088cad7b5eb217493533138196471a8e2180e12f"
Nov 03 23:47:58.024443 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:47:58.024396    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:48:04.042189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.042154    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"]
Nov 03 23:48:04.049898 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.049863    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16a77f-rwhpk"]
Nov 03 23:48:04.209525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.209482    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"]
Nov 03 23:48:04.216116 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.216087    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.222060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.222033    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e4892d\""
Nov 03 23:48:04.227548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.227513    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e4892d-dockercfg-vvphs\""
Nov 03 23:48:04.230110 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.229862    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"]
Nov 03 23:48:04.235483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.235440    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"]
Nov 03 23:48:04.265525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.265490    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.387577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.387487    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.387817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.387794    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwbq\" (UniqueName: \"kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq\") pod \"payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.388769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.388141    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.388985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.388966    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfnh\" (UniqueName: \"kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.408001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.407968    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"]
Nov 03 23:48:04.490813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.490779    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.490988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.490841    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfnh\" (UniqueName: \"kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.490988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.490900    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.490988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.490921    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwbq\" (UniqueName: \"kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq\") pod \"payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.491562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.491521    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.491763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.491725    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.507158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.507127    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfnh\" (UniqueName: \"kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.507711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.507691    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwbq\" (UniqueName: \"kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq\") pod \"payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.542039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.542004    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:04.610698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.610415    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:04.861566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.861528    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"]
Nov 03 23:48:04.861846 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:48:04.861751    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd934e1b5_5031_4075_96f3_70728736b37f.slice/crio-aebc5e8c3c8265b0277072b953b94def52177dd9803ee0b60194b2c4bda5085d WatchSource:0}: Error finding container aebc5e8c3c8265b0277072b953b94def52177dd9803ee0b60194b2c4bda5085d: Status 404 returned error can't find the container with id aebc5e8c3c8265b0277072b953b94def52177dd9803ee0b60194b2c4bda5085d
Nov 03 23:48:04.927773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:04.927732    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"]
Nov 03 23:48:04.928208 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:48:04.928175    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e409d6a_dd66_40db_bec0_8983076a8d4a.slice/crio-04f6035f80c30b011f218a128ba407e39e58bd7bb4570e074601c2b29b3afc5f WatchSource:0}: Error finding container 04f6035f80c30b011f218a128ba407e39e58bd7bb4570e074601c2b29b3afc5f: Status 404 returned error can't find the container with id 04f6035f80c30b011f218a128ba407e39e58bd7bb4570e074601c2b29b3afc5f
Nov 03 23:48:05.020049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:05.020014    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df2221d-80a4-4bbd-8229-77c51b54f3bc" path="/var/lib/kubelet/pods/4df2221d-80a4-4bbd-8229-77c51b54f3bc/volumes"
Nov 03 23:48:05.096358 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:05.096328    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:48:05.176431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:05.176333    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27" event={"ID":"d934e1b5-5031-4075-96f3-70728736b37f","Type":"ContainerStarted","Data":"aebc5e8c3c8265b0277072b953b94def52177dd9803ee0b60194b2c4bda5085d"}
Nov 03 23:48:05.177698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:05.177668    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf" event={"ID":"8e409d6a-dd66-40db-bec0-8983076a8d4a","Type":"ContainerStarted","Data":"04f6035f80c30b011f218a128ba407e39e58bd7bb4570e074601c2b29b3afc5f"}
Nov 03 23:48:07.033253 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.033213    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"]
Nov 03 23:48:07.037670 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.037643    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8a9756-mvhcb"]
Nov 03 23:48:07.206778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.203386    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27" event={"ID":"d934e1b5-5031-4075-96f3-70728736b37f","Type":"ContainerStarted","Data":"1be397144fedb6f1c3ec49109c2a04d923a19c25d74373516a47513b61c0fbee"}
Nov 03 23:48:07.206967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.206809    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf" event={"ID":"8e409d6a-dd66-40db-bec0-8983076a8d4a","Type":"ContainerStarted","Data":"ade18943291bccb12f4b67278cf52c52a7b24916a22859cdfa4561e67bba9875"}
Nov 03 23:48:07.206967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.206859    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:07.206967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.206883    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:07.225838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.225788    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27" podStartSLOduration=1.27962828 podStartE2EDuration="3.225773847s" totalImagesPullingTime="1.946145567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:07.224925068 +0000 UTC m=+1152.324525205" watchObservedRunningTime="2026-11-03 23:48:07.225773847 +0000 UTC m=+1152.325373993"
Nov 03 23:48:07.255790 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:07.255713    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf" podStartSLOduration=1.18373929 podStartE2EDuration="3.255701734s" totalImagesPullingTime="2.071962444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:07.251860381 +0000 UTC m=+1152.351460546" watchObservedRunningTime="2026-11-03 23:48:07.255701734 +0000 UTC m=+1152.355301878"
Nov 03 23:48:09.019711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:09.019673    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715ba7d1-4d00-42b2-a217-604b9963bef8" path="/var/lib/kubelet/pods/715ba7d1-4d00-42b2-a217-604b9963bef8/volumes"
Nov 03 23:48:10.662761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.662386    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"]
Nov 03 23:48:10.793896 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.718978    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"]
Nov 03 23:48:10.793896 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.719100    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:10.818535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.794469    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9bf431\""
Nov 03 23:48:10.818535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.795344    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9bf431-dockercfg-xjq2j\""
Nov 03 23:48:10.818535 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.817797    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"]
Nov 03 23:48:10.832871 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.830996    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlp6m\" (UniqueName: \"kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m\") pod \"payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:10.832871 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.831251    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:10.883174 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.883144    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:10.938780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.938726    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:10.939083 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.939049    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:10.942099 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.942060    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvprh\" (UniqueName: \"kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:10.942548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.942525    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlp6m\" (UniqueName: \"kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m\") pod \"payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:10.943266 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:10.940209    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:11.098772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.074791    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:11.099389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.099350    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvprh\" (UniqueName: \"kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:11.143809 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.128111    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:11.183757 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.181400    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlp6m\" (UniqueName: \"kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m\") pod \"payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:11.272892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.272806    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"]
Nov 03 23:48:11.333053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.333023    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvprh\" (UniqueName: \"kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:11.447701 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.447667    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:11.502910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.502872    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:11.969573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:11.969529    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.332249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.289862    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.335817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.333939    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 23:48:12.371811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.345871    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:12.371811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.345950    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.445623 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.439283    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-d574r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:12.445623 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.439378    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.671109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.671015    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"]
Nov 03 23:48:12.717098 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.716145    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 23:48:12.717425 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:48:12.717357    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14b2c07_8321_4269_9b31_53dba3a5a54b.slice/crio-d0220a6cd2d2d0527a02683586f1b25aa528c97cfb734e7c0cac7be24773d380 WatchSource:0}: Error finding container d0220a6cd2d2d0527a02683586f1b25aa528c97cfb734e7c0cac7be24773d380: Status 404 returned error can't find the container with id d0220a6cd2d2d0527a02683586f1b25aa528c97cfb734e7c0cac7be24773d380
Nov 03 23:48:12.980350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:12.979903    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"]
Nov 03 23:48:13.599235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.598052    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm" event={"ID":"e14b2c07-8321-4269-9b31-53dba3a5a54b","Type":"ContainerStarted","Data":"6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154"}
Nov 03 23:48:13.599235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.598120    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm" event={"ID":"e14b2c07-8321-4269-9b31-53dba3a5a54b","Type":"ContainerStarted","Data":"d0220a6cd2d2d0527a02683586f1b25aa528c97cfb734e7c0cac7be24773d380"}
Nov 03 23:48:13.599235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.599165    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:13.601881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.601855    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd" event={"ID":"e860ce2d-1755-43cd-8b32-d41f442b620c","Type":"ContainerStarted","Data":"ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c"}
Nov 03 23:48:13.602078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.602060    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd" event={"ID":"e860ce2d-1755-43cd-8b32-d41f442b620c","Type":"ContainerStarted","Data":"c7408d0fb4486135b66b615eef19f1f3ba9c8178ef3e9818da6ee58d69d5b911"}
Nov 03 23:48:13.602910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.602890    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:13.654598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.654539    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm" podStartSLOduration=3.654523858 podStartE2EDuration="3.654523858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:13.650094141 +0000 UTC m=+1158.749694283" watchObservedRunningTime="2026-11-03 23:48:13.654523858 +0000 UTC m=+1158.754124002"
Nov 03 23:48:13.681646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:13.681598    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd" podStartSLOduration=3.681584053 podStartE2EDuration="3.681584053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:13.677321819 +0000 UTC m=+1158.776921962" watchObservedRunningTime="2026-11-03 23:48:13.681584053 +0000 UTC m=+1158.781184229"
Nov 03 23:48:18.252227 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:18.252194    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:48:18.252842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:18.252337    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:48:20.035892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:20.035855    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"]
Nov 03 23:48:20.040557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:20.040528    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6526719a-q247p"]
Nov 03 23:48:21.019399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:21.019368    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b56aa20-a5ae-42b9-b82d-d886c938c2f8" path="/var/lib/kubelet/pods/6b56aa20-a5ae-42b9-b82d-d886c938c2f8/volumes"
Nov 03 23:48:25.628955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:25.628925    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:48:25.629436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:25.629097    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:48:28.026633 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:48:28.026595    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:48:34.038100 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:34.038067    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"]
Nov 03 23:48:34.061674 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:34.061612    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46b7b73-jgkfk"]
Nov 03 23:48:35.044094 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:35.044054    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8165cf-bc21-4c4c-9126-450c99696f6b" path="/var/lib/kubelet/pods/cf8165cf-bc21-4c4c-9126-450c99696f6b/volumes"
Nov 03 23:48:38.058779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:38.056800    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"]
Nov 03 23:48:38.061433 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:38.061402    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a36547-ldsjw"]
Nov 03 23:48:39.020161 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:39.020123    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd020c42-b2df-4082-98f5-799fca16f397" path="/var/lib/kubelet/pods/cd020c42-b2df-4082-98f5-799fca16f397/volumes"
Nov 03 23:48:40.322288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.322242    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"]
Nov 03 23:48:40.332487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.331855    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"]
Nov 03 23:48:40.332487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.331947    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.346493 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.346468    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8cfc3c\""
Nov 03 23:48:40.346775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.346723    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8cfc3c-dockercfg-vws7g\""
Nov 03 23:48:40.417677 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.417636    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq22g\" (UniqueName: \"kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.417879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.417755    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.533758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.529799    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq22g\" (UniqueName: \"kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.546035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.545993    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.547484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.547459    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.585346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.585264    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq22g\" (UniqueName: \"kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:40.655981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:40.655949    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:41.023834 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:48:41.022330    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4101bac_ca3c_43d1_b7a2_89af28641686.slice/crio-946505db41e983d73eb330b834b580636df076cedfa958991e327bcc5d39812f WatchSource:0}: Error finding container 946505db41e983d73eb330b834b580636df076cedfa958991e327bcc5d39812f: Status 404 returned error can't find the container with id 946505db41e983d73eb330b834b580636df076cedfa958991e327bcc5d39812f
Nov 03 23:48:41.029834 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:41.028760    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"]
Nov 03 23:48:42.017839 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:42.017795    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx" event={"ID":"f4101bac-ca3c-43d1-b7a2-89af28641686","Type":"ContainerStarted","Data":"ee2a9e6fa1f175902decd116066953a2a39af00b6883246af4285d5b5dfa45b6"}
Nov 03 23:48:42.018331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:42.017866    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx" event={"ID":"f4101bac-ca3c-43d1-b7a2-89af28641686","Type":"ContainerStarted","Data":"946505db41e983d73eb330b834b580636df076cedfa958991e327bcc5d39812f"}
Nov 03 23:48:42.018331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:42.018005    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:42.047884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:42.047822    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx" podStartSLOduration=2.047803568 podStartE2EDuration="2.047803568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:42.042174959 +0000 UTC m=+1187.141775114" watchObservedRunningTime="2026-11-03 23:48:42.047803568 +0000 UTC m=+1187.147403713"
Nov 03 23:48:48.039182 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.039143    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"]
Nov 03 23:48:48.053873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.053835    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b60f24-mhqtx"]
Nov 03 23:48:48.072390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.072361    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"]
Nov 03 23:48:48.077498 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.077473    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.088008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.087979    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-88f34dbe-openshift-default-dockercfg-nhpxr\""
Nov 03 23:48:48.098647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.098616    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"]
Nov 03 23:48:48.159984 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.159946    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzv8\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160054    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160083    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160220    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160367 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160330    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160442 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160370    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160442 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160405    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160522    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.160602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.160583    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.261976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.261938    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.261998    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262035    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262075    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262170    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262402    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzv8\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262467    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262473    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262516    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262529    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262592    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262902 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262853    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.262902 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262888    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.263046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.262962    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.264363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.264341    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.265140 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.265119    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.275897 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.275875    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.276022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.276005    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzv8\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.405633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.400803    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:48.712611 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:48:48.712581    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f725dc_341e_4ce2_a4dc_4385e6d8afab.slice/crio-115c409ea3f21c0b06bc4b1e7c80ef095a4d62f4d2b43ec843516d910fc82209 WatchSource:0}: Error finding container 115c409ea3f21c0b06bc4b1e7c80ef095a4d62f4d2b43ec843516d910fc82209: Status 404 returned error can't find the container with id 115c409ea3f21c0b06bc4b1e7c80ef095a4d62f4d2b43ec843516d910fc82209
Nov 03 23:48:48.713843 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.713817    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"]
Nov 03 23:48:48.715903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.715860    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:48:48.716001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.715939    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:48:48.716001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:48.715982    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:48:49.022641 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.022534    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a97108-5998-4704-bf82-d4c0067cdac8" path="/var/lib/kubelet/pods/23a97108-5998-4704-bf82-d4c0067cdac8/volumes"
Nov 03 23:48:49.084290 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.084240    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" event={"ID":"54f725dc-341e-4ce2-a4dc-4385e6d8afab","Type":"ContainerStarted","Data":"11f21774d20e2880f8b4e45b47879414a65f76c956dfd1456f38bb6f147bef60"}
Nov 03 23:48:49.084697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.084316    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" event={"ID":"54f725dc-341e-4ce2-a4dc-4385e6d8afab","Type":"ContainerStarted","Data":"115c409ea3f21c0b06bc4b1e7c80ef095a4d62f4d2b43ec843516d910fc82209"}
Nov 03 23:48:49.110691 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.110634    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podStartSLOduration=1.11061934 podStartE2EDuration="1.11061934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:49.107980047 +0000 UTC m=+1194.207580192" watchObservedRunningTime="2026-11-03 23:48:49.11061934 +0000 UTC m=+1194.210219541"
Nov 03 23:48:49.404198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.404108    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:49.408906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.408867    5074 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:48:49.409190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:49.409161    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 03 23:48:50.415765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.415176    5074 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:48:50.415765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.415243    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 03 23:48:50.537524 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.537491    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7308-9998ca"]
Nov 03 23:48:50.542297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.542271    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:48:50.547769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.547451    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hxqlx\""
Nov 03 23:48:50.547769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.547632    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7308-9998ca"]
Nov 03 23:48:50.624887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.624847    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mvn\" (UniqueName: \"kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn\") pod \"test-curl-7308-9998ca\" (UID: \"5253b38a-adad-45eb-bafd-05b2179d2403\") " pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:48:50.741923 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.741876    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mvn\" (UniqueName: \"kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn\") pod \"test-curl-7308-9998ca\" (UID: \"5253b38a-adad-45eb-bafd-05b2179d2403\") " pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:48:50.768771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.767683    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mvn\" (UniqueName: \"kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn\") pod \"test-curl-7308-9998ca\" (UID: \"5253b38a-adad-45eb-bafd-05b2179d2403\") " pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:48:50.859782 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:50.859373    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:48:51.287923 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:51.287890    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7308-9998ca"]
Nov 03 23:48:51.292879 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:48:51.292852    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5253b38a_adad_45eb_bafd_05b2179d2403.slice/crio-2beb75c527299d01a71006d1ad458aba1fc4e002a329d464e2d842a8ae6eb76d WatchSource:0}: Error finding container 2beb75c527299d01a71006d1ad458aba1fc4e002a329d464e2d842a8ae6eb76d: Status 404 returned error can't find the container with id 2beb75c527299d01a71006d1ad458aba1fc4e002a329d464e2d842a8ae6eb76d
Nov 03 23:48:51.421843 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:51.403099    5074 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:48:51.421843 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:51.403160    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 03 23:48:52.157335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:52.157293    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7308-9998ca" event={"ID":"5253b38a-adad-45eb-bafd-05b2179d2403","Type":"ContainerStarted","Data":"2beb75c527299d01a71006d1ad458aba1fc4e002a329d464e2d842a8ae6eb76d"}
Nov 03 23:48:52.409652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:52.409153    5074 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:48:52.409652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:52.409214    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 03 23:48:53.089670 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:53.089639    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:48:53.422655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:53.422143    5074 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:48:53.422655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:53.422202    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 03 23:48:54.423769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:54.422250    5074 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:48:54.423769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:54.422316    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 03 23:48:55.267298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.267186    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7308-9998ca" event={"ID":"5253b38a-adad-45eb-bafd-05b2179d2403","Type":"ContainerStarted","Data":"03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96"}
Nov 03 23:48:55.291493 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.291435    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7308-9998ca" podStartSLOduration=1.6217833289999999 podStartE2EDuration="5.291417004s" totalImagesPullingTime="3.669633675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:55.28996566 +0000 UTC m=+1200.389565804" watchObservedRunningTime="2026-11-03 23:48:55.291417004 +0000 UTC m=+1200.391017149"
Nov 03 23:48:55.331039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.331010    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:48:55.340305 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.340278    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:48:55.434296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.434259    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:55.434808 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.434779    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:55.435982 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.435959    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:48:55.550785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.550698    5074 scope.go:122] "RemoveContainer" containerID="fb1214939d643771630d3dd9beabd9d41cf2cd6ffad64d3fc7fa8b16551ae99f"
Nov 03 23:48:55.571329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.571302    5074 scope.go:122] "RemoveContainer" containerID="4992ce638361b4f733b33c5b2c7aa50815e3ad78f90503502fcdde0fcf913ded"
Nov 03 23:48:55.591754 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:55.591714    5074 scope.go:122] "RemoveContainer" containerID="64b73f19ec402026228501226a9011aba2314ed3469d2d4881ee8646e675a585"
Nov 03 23:48:56.081655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:56.081608    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7308-9998ca"]
Nov 03 23:48:56.087342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:56.087303    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7308-9998ca"]
Nov 03 23:48:57.286541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:57.286496    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7308-9998ca" podUID="5253b38a-adad-45eb-bafd-05b2179d2403" containerName="test-curl-7308-9998ca" containerID="cri-o://03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96" gracePeriod=2
Nov 03 23:48:58.025862 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:48:58.025813    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:48:59.490611 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:59.490588    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:48:59.546041 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:59.545939    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn\" (UniqueName: \"kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn\") pod \"5253b38a-adad-45eb-bafd-05b2179d2403\" (UID: \"5253b38a-adad-45eb-bafd-05b2179d2403\") "
Nov 03 23:48:59.548111 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:59.548087    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn" pod "5253b38a-adad-45eb-bafd-05b2179d2403" (UID: "5253b38a-adad-45eb-bafd-05b2179d2403"). InnerVolumeSpecName "kube-api-access-87mvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:48:59.647694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:48:59.647655    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87mvn\" (UniqueName: \"kubernetes.io/projected/5253b38a-adad-45eb-bafd-05b2179d2403-kube-api-access-87mvn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:00.032530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.032497    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"]
Nov 03 23:49:00.037001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.036975    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cacd2ccf-zkpbx"]
Nov 03 23:49:00.318504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.318404    5074 generic.go:356] "Generic (PLEG): container finished" podID="5253b38a-adad-45eb-bafd-05b2179d2403" containerID="03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96" exitCode=137
Nov 03 23:49:00.318504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.318459    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:49:00.318729 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.318515    5074 scope.go:122] "RemoveContainer" containerID="03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96"
Nov 03 23:49:00.330588 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:00.330556    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7308-9998ca\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="5253b38a-adad-45eb-bafd-05b2179d2403" pod="opendatahub/test-curl-7308-9998ca"
Nov 03 23:49:00.332107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.332090    5074 scope.go:122] "RemoveContainer" containerID="03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96"
Nov 03 23:49:00.332404 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:00.332386    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96\": container with ID starting with 03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96 not found: ID does not exist" containerID="03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96"
Nov 03 23:49:00.332457 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:00.332414    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96"} err="failed to get container status \"03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96\": rpc error: code = NotFound desc = could not find container \"03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96\": container with ID starting with 03329b10e0dbc44fd18d2fcd0272042d69b36cb3f25acdd0f4533fa54024dc96 not found: ID does not exist"
Nov 03 23:49:01.020584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:01.020550    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5253b38a-adad-45eb-bafd-05b2179d2403" path="/var/lib/kubelet/pods/5253b38a-adad-45eb-bafd-05b2179d2403/volumes"
Nov 03 23:49:01.021114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:01.021029    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45de119-37f8-417f-8e39-603b4c7bdfeb" path="/var/lib/kubelet/pods/a45de119-37f8-417f-8e39-603b4c7bdfeb/volumes"
Nov 03 23:49:04.184143 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.184099    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"]
Nov 03 23:49:04.507200 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.507112    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"]
Nov 03 23:49:04.507943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.507923    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5253b38a-adad-45eb-bafd-05b2179d2403" containerName="test-curl-7308-9998ca"
Nov 03 23:49:04.507943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.507942    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5253b38a-adad-45eb-bafd-05b2179d2403" containerName="test-curl-7308-9998ca"
Nov 03 23:49:04.508125 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.508053    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5253b38a-adad-45eb-bafd-05b2179d2403" containerName="test-curl-7308-9998ca"
Nov 03 23:49:04.513310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.513283    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.524776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.524753    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c887c7\""
Nov 03 23:49:04.525650 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.525605    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c887c7-dockercfg-7l9rb\""
Nov 03 23:49:04.533035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.532180    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"]
Nov 03 23:49:04.556786 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.556754    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"]
Nov 03 23:49:04.569983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.569956    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.572306 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.572279    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"]
Nov 03 23:49:04.646769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.641278    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmhp\" (UniqueName: \"kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.646769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.641365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.751609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.751162    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmhp\" (UniqueName: \"kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.751609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.751267    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.751609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.751352    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzqrv\" (UniqueName: \"kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv\") pod \"payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.751609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.751385    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.772815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.752869    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.880043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.863893    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzqrv\" (UniqueName: \"kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv\") pod \"payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.880043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.863946    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.880043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.864918    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.897231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.882464    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmhp\" (UniqueName: \"kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:04.897231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.884546    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzqrv\" (UniqueName: \"kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv\") pod \"payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:04.905983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:04.898831    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:05.138346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:05.138313    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:05.316365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:05.315410    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"]
Nov 03 23:49:05.392129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:05.392094    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2" event={"ID":"f209157f-4001-47b8-bf92-293790d63df4","Type":"ContainerStarted","Data":"b29bf1b53679c0430dbc1f9b272c3734003b90d3595291ca21436a7a6ea83330"}
Nov 03 23:49:05.461944 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:49:05.461904    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27468ed_ec84_419d_8e12_a5bf8e3e4e80.slice/crio-fbc55dfbc00f72dcf8da0b181b37da265e5c352d5312b8d211cf576f714cd786 WatchSource:0}: Error finding container fbc55dfbc00f72dcf8da0b181b37da265e5c352d5312b8d211cf576f714cd786: Status 404 returned error can't find the container with id fbc55dfbc00f72dcf8da0b181b37da265e5c352d5312b8d211cf576f714cd786
Nov 03 23:49:05.462629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:05.462604    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"]
Nov 03 23:49:05.778437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:05.778397    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"]
Nov 03 23:49:06.404884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.404832    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2" event={"ID":"f209157f-4001-47b8-bf92-293790d63df4","Type":"ContainerStarted","Data":"21a95a93bdd4f59f8c8acdcf66a75cb2d651f9c4220dcab6486b41766ff6cfb9"}
Nov 03 23:49:06.405368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.405125    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:06.407634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.407606    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp" event={"ID":"d27468ed-ec84-419d-8e12-a5bf8e3e4e80","Type":"ContainerStarted","Data":"45245ecfa62fc0d854c66063afbdc0ae5bc5e02582771d5c2c0e910bff87b726"}
Nov 03 23:49:06.407726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.407714    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp" event={"ID":"d27468ed-ec84-419d-8e12-a5bf8e3e4e80","Type":"ContainerStarted","Data":"fbc55dfbc00f72dcf8da0b181b37da265e5c352d5312b8d211cf576f714cd786"}
Nov 03 23:49:06.408681 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.408662    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:06.434975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.434913    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2" podStartSLOduration=2.434894757 podStartE2EDuration="2.434894757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:06.431888989 +0000 UTC m=+1211.531489133" watchObservedRunningTime="2026-11-03 23:49:06.434894757 +0000 UTC m=+1211.534494906"
Nov 03 23:49:06.456416 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:06.456351    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp" podStartSLOduration=2.456331583 podStartE2EDuration="2.456331583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:06.452132158 +0000 UTC m=+1211.551732297" watchObservedRunningTime="2026-11-03 23:49:06.456331583 +0000 UTC m=+1211.555931735"
Nov 03 23:49:09.187100 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.187060    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx" podUID="f4101bac-ca3c-43d1-b7a2-89af28641686" containerName="payload-pre-processing" containerID="cri-o://ee2a9e6fa1f175902decd116066953a2a39af00b6883246af4285d5b5dfa45b6" gracePeriod=25
Nov 03 23:49:09.511315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.511279    5074 generic.go:356] "Generic (PLEG): container finished" podID="f4101bac-ca3c-43d1-b7a2-89af28641686" containerID="ee2a9e6fa1f175902decd116066953a2a39af00b6883246af4285d5b5dfa45b6" exitCode=0
Nov 03 23:49:09.511431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.511352    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx" event={"ID":"f4101bac-ca3c-43d1-b7a2-89af28641686","Type":"ContainerDied","Data":"ee2a9e6fa1f175902decd116066953a2a39af00b6883246af4285d5b5dfa45b6"}
Nov 03 23:49:09.556880 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.556846    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:49:09.698989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.698951    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume\") pod \"f4101bac-ca3c-43d1-b7a2-89af28641686\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") "
Nov 03 23:49:09.699158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.699068    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g\" (UniqueName: \"kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g\") pod \"f4101bac-ca3c-43d1-b7a2-89af28641686\" (UID: \"f4101bac-ca3c-43d1-b7a2-89af28641686\") "
Nov 03 23:49:09.699315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.699294    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume" pod "f4101bac-ca3c-43d1-b7a2-89af28641686" (UID: "f4101bac-ca3c-43d1-b7a2-89af28641686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:09.699539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.699525    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4101bac-ca3c-43d1-b7a2-89af28641686-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:09.701229 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.701203    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g" pod "f4101bac-ca3c-43d1-b7a2-89af28641686" (UID: "f4101bac-ca3c-43d1-b7a2-89af28641686"). InnerVolumeSpecName "kube-api-access-rq22g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:09.800982 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:09.800905    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq22g\" (UniqueName: \"kubernetes.io/projected/f4101bac-ca3c-43d1-b7a2-89af28641686-kube-api-access-rq22g\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:10.522546 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:10.522514    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"
Nov 03 23:49:10.523064 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:10.522553    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx" event={"ID":"f4101bac-ca3c-43d1-b7a2-89af28641686","Type":"ContainerDied","Data":"946505db41e983d73eb330b834b580636df076cedfa958991e327bcc5d39812f"}
Nov 03 23:49:10.523064 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:10.522620    5074 scope.go:122] "RemoveContainer" containerID="ee2a9e6fa1f175902decd116066953a2a39af00b6883246af4285d5b5dfa45b6"
Nov 03 23:49:10.553254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:10.553227    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"]
Nov 03 23:49:10.557229 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:10.557207    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-66bb7d65b7-56jnx"]
Nov 03 23:49:10.779254 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:10.779165    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr" podUID="4ad039f7-d961-4068-bc75-d34b87a999a9" containerName="payload-processing" containerID="cri-o://ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578" gracePeriod=25
Nov 03 23:49:11.028049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.028007    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4101bac-ca3c-43d1-b7a2-89af28641686" path="/var/lib/kubelet/pods/f4101bac-ca3c-43d1-b7a2-89af28641686/volumes"
Nov 03 23:49:11.081390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.081363    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:49:11.225904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.225875    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx\" (UniqueName: \"kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx\") pod \"4ad039f7-d961-4068-bc75-d34b87a999a9\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") "
Nov 03 23:49:11.226108 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.226089    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume\") pod \"4ad039f7-d961-4068-bc75-d34b87a999a9\" (UID: \"4ad039f7-d961-4068-bc75-d34b87a999a9\") "
Nov 03 23:49:11.226519 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.226488    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume" pod "4ad039f7-d961-4068-bc75-d34b87a999a9" (UID: "4ad039f7-d961-4068-bc75-d34b87a999a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:11.226856 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.226834    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad039f7-d961-4068-bc75-d34b87a999a9-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:11.228195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.228168    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx" pod "4ad039f7-d961-4068-bc75-d34b87a999a9" (UID: "4ad039f7-d961-4068-bc75-d34b87a999a9"). InnerVolumeSpecName "kube-api-access-ld5lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:11.327904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.327828    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5lx\" (UniqueName: \"kubernetes.io/projected/4ad039f7-d961-4068-bc75-d34b87a999a9-kube-api-access-ld5lx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:11.534436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.534398    5074 generic.go:356] "Generic (PLEG): container finished" podID="4ad039f7-d961-4068-bc75-d34b87a999a9" containerID="ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578" exitCode=0
Nov 03 23:49:11.534918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.534451    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"
Nov 03 23:49:11.534918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.534474    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr" event={"ID":"4ad039f7-d961-4068-bc75-d34b87a999a9","Type":"ContainerDied","Data":"ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578"}
Nov 03 23:49:11.534918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.534541    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr" event={"ID":"4ad039f7-d961-4068-bc75-d34b87a999a9","Type":"ContainerDied","Data":"ee25b089bc4ee30db487dad1d588815f7011ab652e9da3f116d4d4fa4252d354"}
Nov 03 23:49:11.534918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.534571    5074 scope.go:122] "RemoveContainer" containerID="ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578"
Nov 03 23:49:11.549574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.549555    5074 scope.go:122] "RemoveContainer" containerID="ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578"
Nov 03 23:49:11.549932 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:11.549881    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578\": container with ID starting with ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578 not found: ID does not exist" containerID="ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578"
Nov 03 23:49:11.550033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.549947    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578"} err="failed to get container status \"ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578\": rpc error: code = NotFound desc = could not find container \"ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578\": container with ID starting with ac3f46440a141dd0d4bded030a9d49989557a177acc46dab994d4d5204729578 not found: ID does not exist"
Nov 03 23:49:11.564515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.564492    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"]
Nov 03 23:49:11.568328 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:11.568308    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d41841-744c8b765d-7csvr"]
Nov 03 23:49:12.051641 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:12.051603    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"]
Nov 03 23:49:12.058277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:12.058238    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8e5d23-ktsnp"]
Nov 03 23:49:13.019711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:13.019677    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad039f7-d961-4068-bc75-d34b87a999a9" path="/var/lib/kubelet/pods/4ad039f7-d961-4068-bc75-d34b87a999a9/volumes"
Nov 03 23:49:13.020378 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:13.020356    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06" path="/var/lib/kubelet/pods/ec9691bb-e223-43e5-9ac6-bbbc9d2c2e06/volumes"
Nov 03 23:49:15.955720 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.955688    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"]
Nov 03 23:49:15.956397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.956378    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f4101bac-ca3c-43d1-b7a2-89af28641686" containerName="payload-pre-processing"
Nov 03 23:49:15.956453 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.956399    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4101bac-ca3c-43d1-b7a2-89af28641686" containerName="payload-pre-processing"
Nov 03 23:49:15.956453 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.956443    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4ad039f7-d961-4068-bc75-d34b87a999a9" containerName="payload-processing"
Nov 03 23:49:15.956453 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.956451    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad039f7-d961-4068-bc75-d34b87a999a9" containerName="payload-processing"
Nov 03 23:49:15.956568 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.956557    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad039f7-d961-4068-bc75-d34b87a999a9" containerName="payload-processing"
Nov 03 23:49:15.956604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.956574    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4101bac-ca3c-43d1-b7a2-89af28641686" containerName="payload-pre-processing"
Nov 03 23:49:15.961212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.961193    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:15.969565 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.969029    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8cfc3c\""
Nov 03 23:49:15.971112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.971089    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8cfc3c-dockercfg-bvq2d\""
Nov 03 23:49:15.973429 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:15.973398    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"]
Nov 03 23:49:16.121785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.121717    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fc7\" (UniqueName: \"kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7\") pod \"payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.122724 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.122696    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.227035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.226220    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fc7\" (UniqueName: \"kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7\") pod \"payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.227035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.226550    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.229595 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.228770    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.244149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.244117    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fc7\" (UniqueName: \"kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7\") pod \"payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.279329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.279091    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:16.635708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:16.635181    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"]
Nov 03 23:49:17.424437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:17.424407    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:49:17.664434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:17.659009    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f" event={"ID":"74f6c664-c4db-4d8d-9e0e-8cbb23d26358","Type":"ContainerStarted","Data":"db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7"}
Nov 03 23:49:17.664434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:17.659079    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f" event={"ID":"74f6c664-c4db-4d8d-9e0e-8cbb23d26358","Type":"ContainerStarted","Data":"f3e8fbc6d1c6785b16335237a2190b45a3c206390283c9debc98667116dc7636"}
Nov 03 23:49:17.664434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:17.659962    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:17.689934 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:17.689857    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f" podStartSLOduration=2.6898375469999998 podStartE2EDuration="2.689837547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:17.684769404 +0000 UTC m=+1222.784369574" watchObservedRunningTime="2026-11-03 23:49:17.689837547 +0000 UTC m=+1222.789437693"
Nov 03 23:49:18.477169 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:18.477142    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:49:21.037750 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:21.037707    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"]
Nov 03 23:49:21.044112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:21.044084    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1400539-9k96l"]
Nov 03 23:49:23.019975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:23.019944    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f040f79-9279-4e90-8b23-833ddabc12d0" path="/var/lib/kubelet/pods/4f040f79-9279-4e90-8b23-833ddabc12d0/volumes"
Nov 03 23:49:23.376774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:23.376676    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2_f209157f-4001-47b8-bf92-293790d63df4/payload-processing/0.log"
Nov 03 23:49:24.049523 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:24.049488    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"]
Nov 03 23:49:24.054287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:24.054254    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630011a8-9nxzn"]
Nov 03 23:49:25.021301 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:25.021259    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a764d63b-1c3f-4578-9bbe-9b06cb89ba03" path="/var/lib/kubelet/pods/a764d63b-1c3f-4578-9bbe-9b06cb89ba03/volumes"
Nov 03 23:49:28.023718 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:28.023680    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:49:29.030096 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:29.030042    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"]
Nov 03 23:49:29.034714 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:29.034689    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0f8676-jxwvt"]
Nov 03 23:49:29.685028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:29.684988    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:49:31.019821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:31.019791    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596191db-dd36-4d68-bc86-69878b1bfae4" path="/var/lib/kubelet/pods/596191db-dd36-4d68-bc86-69878b1bfae4/volumes"
Nov 03 23:49:33.045832 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:33.045796    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"]
Nov 03 23:49:33.049525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:33.049498    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6ebbc6d-8sq7v"]
Nov 03 23:49:35.056660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:35.056618    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7969f1-2a24-4327-8aae-162a24fc13d3" path="/var/lib/kubelet/pods/2a7969f1-2a24-4327-8aae-162a24fc13d3/volumes"
Nov 03 23:49:44.314231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:44.314030    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"]
Nov 03 23:49:44.316233 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:44.315621    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm" podUID="e14b2c07-8321-4269-9b31-53dba3a5a54b" containerName="payload-processing" containerID="cri-o://6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154" gracePeriod=30
Nov 03 23:49:44.326975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:44.326952    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"]
Nov 03 23:49:44.327637 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:44.327377    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd" podUID="e860ce2d-1755-43cd-8b32-d41f442b620c" containerName="payload-pre-processing" containerID="cri-o://ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c" gracePeriod=30
Nov 03 23:49:44.962112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:44.962081    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:49:44.967270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:44.966863    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:49:45.046784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.046749    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"]
Nov 03 23:49:45.052136 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.052107    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d0b92-bvg4d"]
Nov 03 23:49:45.054335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.054308    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume\") pod \"e14b2c07-8321-4269-9b31-53dba3a5a54b\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") "
Nov 03 23:49:45.054466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.054342    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume\") pod \"e860ce2d-1755-43cd-8b32-d41f442b620c\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") "
Nov 03 23:49:45.054466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.054414    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m\" (UniqueName: \"kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m\") pod \"e14b2c07-8321-4269-9b31-53dba3a5a54b\" (UID: \"e14b2c07-8321-4269-9b31-53dba3a5a54b\") "
Nov 03 23:49:45.054466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.054446    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh\" (UniqueName: \"kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh\") pod \"e860ce2d-1755-43cd-8b32-d41f442b620c\" (UID: \"e860ce2d-1755-43cd-8b32-d41f442b620c\") "
Nov 03 23:49:45.055050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.055022    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume" pod "e860ce2d-1755-43cd-8b32-d41f442b620c" (UID: "e860ce2d-1755-43cd-8b32-d41f442b620c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:45.055236 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.055209    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume" pod "e14b2c07-8321-4269-9b31-53dba3a5a54b" (UID: "e14b2c07-8321-4269-9b31-53dba3a5a54b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:45.056024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.056002    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14b2c07-8321-4269-9b31-53dba3a5a54b-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.056024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.056025    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e860ce2d-1755-43cd-8b32-d41f442b620c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.057222 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.057198    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh" pod "e860ce2d-1755-43cd-8b32-d41f442b620c" (UID: "e860ce2d-1755-43cd-8b32-d41f442b620c"). InnerVolumeSpecName "kube-api-access-mvprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:45.060007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.059976    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m" pod "e14b2c07-8321-4269-9b31-53dba3a5a54b" (UID: "e14b2c07-8321-4269-9b31-53dba3a5a54b"). InnerVolumeSpecName "kube-api-access-zlp6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:45.094771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.094718    5074 generic.go:356] "Generic (PLEG): container finished" podID="e14b2c07-8321-4269-9b31-53dba3a5a54b" containerID="6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154" exitCode=0
Nov 03 23:49:45.094961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.094787    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"
Nov 03 23:49:45.094961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.094804    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm" event={"ID":"e14b2c07-8321-4269-9b31-53dba3a5a54b","Type":"ContainerDied","Data":"6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154"}
Nov 03 23:49:45.094961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.094868    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm" event={"ID":"e14b2c07-8321-4269-9b31-53dba3a5a54b","Type":"ContainerDied","Data":"d0220a6cd2d2d0527a02683586f1b25aa528c97cfb734e7c0cac7be24773d380"}
Nov 03 23:49:45.094961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.094898    5074 scope.go:122] "RemoveContainer" containerID="6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154"
Nov 03 23:49:45.103788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.101822    5074 generic.go:356] "Generic (PLEG): container finished" podID="e860ce2d-1755-43cd-8b32-d41f442b620c" containerID="ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c" exitCode=0
Nov 03 23:49:45.103788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.102035    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd" event={"ID":"e860ce2d-1755-43cd-8b32-d41f442b620c","Type":"ContainerDied","Data":"ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c"}
Nov 03 23:49:45.103788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.102085    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd" event={"ID":"e860ce2d-1755-43cd-8b32-d41f442b620c","Type":"ContainerDied","Data":"c7408d0fb4486135b66b615eef19f1f3ba9c8178ef3e9818da6ee58d69d5b911"}
Nov 03 23:49:45.103788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.102149    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"
Nov 03 23:49:45.110885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.110866    5074 scope.go:122] "RemoveContainer" containerID="6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154"
Nov 03 23:49:45.111189 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:45.111171    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154\": container with ID starting with 6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154 not found: ID does not exist" containerID="6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154"
Nov 03 23:49:45.111265 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.111197    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154"} err="failed to get container status \"6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154\": rpc error: code = NotFound desc = could not find container \"6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154\": container with ID starting with 6c334c3b7eba18fac98bfe99def03ab932482e884f1844f2bcc12b63f9a31154 not found: ID does not exist"
Nov 03 23:49:45.111265 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.111214    5074 scope.go:122] "RemoveContainer" containerID="ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c"
Nov 03 23:49:45.125759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.125725    5074 scope.go:122] "RemoveContainer" containerID="ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c"
Nov 03 23:49:45.126135 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:45.126110    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c\": container with ID starting with ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c not found: ID does not exist" containerID="ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c"
Nov 03 23:49:45.126200 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.126148    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c"} err="failed to get container status \"ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c\": rpc error: code = NotFound desc = could not find container \"ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c\": container with ID starting with ff75ca45d372cec44dab0bffa1c1298d56c22d026c8f0d776965f5fab29f179c not found: ID does not exist"
Nov 03 23:49:45.138025 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.137999    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"]
Nov 03 23:49:45.144102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.144041    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-5fd9b9f57b-jnhsm"]
Nov 03 23:49:45.156520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.156495    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"]
Nov 03 23:49:45.157298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.157276    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlp6m\" (UniqueName: \"kubernetes.io/projected/e14b2c07-8321-4269-9b31-53dba3a5a54b-kube-api-access-zlp6m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.157381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.157301    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvprh\" (UniqueName: \"kubernetes.io/projected/e860ce2d-1755-43cd-8b32-d41f442b620c-kube-api-access-mvprh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.168280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:45.168253    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-5d44fd5587-bv9dd"]
Nov 03 23:49:47.021975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:47.021940    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef2abca-9f4b-42e6-ae8e-13bf3c3849ea" path="/var/lib/kubelet/pods/aef2abca-9f4b-42e6-ae8e-13bf3c3849ea/volumes"
Nov 03 23:49:47.022558 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:47.022537    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14b2c07-8321-4269-9b31-53dba3a5a54b" path="/var/lib/kubelet/pods/e14b2c07-8321-4269-9b31-53dba3a5a54b/volumes"
Nov 03 23:49:47.022991 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:47.022971    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e860ce2d-1755-43cd-8b32-d41f442b620c" path="/var/lib/kubelet/pods/e860ce2d-1755-43cd-8b32-d41f442b620c/volumes"
Nov 03 23:49:48.050600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:48.050571    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"]
Nov 03 23:49:48.054060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:48.054039    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0630113b-4ws5t"]
Nov 03 23:49:49.022472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:49.022437    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417e1195-64de-4597-96d4-b8ef49c2bc95" path="/var/lib/kubelet/pods/417e1195-64de-4597-96d4-b8ef49c2bc95/volumes"
Nov 03 23:49:55.036944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.036912    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"]
Nov 03 23:49:55.048396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.048358    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4817a2-qmnl4"]
Nov 03 23:49:55.657222 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.657191    5074 scope.go:122] "RemoveContainer" containerID="63d5d7bab08ce773d6a5fbcb4f78e76a8555aeeea61892e645962e758ecbde9a"
Nov 03 23:49:55.669150 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.669124    5074 scope.go:122] "RemoveContainer" containerID="c12945f478e29842baaaea63583fa12b38b2de825a1e6af75aa448f591450b11"
Nov 03 23:49:55.679186 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.679160    5074 scope.go:122] "RemoveContainer" containerID="89b4c4ca3a29addeecfd8c4f4183c79db3cef7996edc16ee83ce630829cf413f"
Nov 03 23:49:55.689059 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.689036    5074 scope.go:122] "RemoveContainer" containerID="c34b125c2286c51dab3df35d7e3f57c387061e1f5a0001f1cb3b50c59a505c9c"
Nov 03 23:49:55.698583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.698564    5074 scope.go:122] "RemoveContainer" containerID="740a265fb7ff441d105883749f073872314f10d18b0661aeb9f2221f5a49f95c"
Nov 03 23:49:55.707657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:55.707636    5074 scope.go:122] "RemoveContainer" containerID="1c39c64cf6a42e9ca09674815ccf9aa45d6f2c100db0fb5313881b4c9a56384e"
Nov 03 23:49:57.034761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:57.026481    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f92671-f4b0-4fd0-a5de-fd6a273cd295" path="/var/lib/kubelet/pods/97f92671-f4b0-4fd0-a5de-fd6a273cd295/volumes"
Nov 03 23:49:57.044076 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:57.044044    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"]
Nov 03 23:49:57.049948 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:57.049909    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e19ddf-9xc2l"]
Nov 03 23:49:58.024134 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:49:58.024099    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:49:58.031792 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:58.031762    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"]
Nov 03 23:49:58.035256 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:58.035234    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279ff644-dk9kw"]
Nov 03 23:49:59.022208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:59.022162    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa992b3-7a8d-4217-8e38-cf9ae79857bf" path="/var/lib/kubelet/pods/3fa992b3-7a8d-4217-8e38-cf9ae79857bf/volumes"
Nov 03 23:49:59.022936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:49:59.022918    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7801f3b4-01cb-42f2-99d9-71d01cec1d40" path="/var/lib/kubelet/pods/7801f3b4-01cb-42f2-99d9-71d01cec1d40/volumes"
Nov 03 23:50:00.098796 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:00.098034    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"]
Nov 03 23:50:00.098796 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:00.098286    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy" containerID="cri-o://11f21774d20e2880f8b4e45b47879414a65f76c956dfd1456f38bb6f147bef60" gracePeriod=30
Nov 03 23:50:05.387431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.387402    5074 generic.go:356] "Generic (PLEG): container finished" podID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerID="11f21774d20e2880f8b4e45b47879414a65f76c956dfd1456f38bb6f147bef60" exitCode=0
Nov 03 23:50:05.387780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.387479    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" event={"ID":"54f725dc-341e-4ce2-a4dc-4385e6d8afab","Type":"ContainerDied","Data":"11f21774d20e2880f8b4e45b47879414a65f76c956dfd1456f38bb6f147bef60"}
Nov 03 23:50:05.387780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.387532    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf" event={"ID":"54f725dc-341e-4ce2-a4dc-4385e6d8afab","Type":"ContainerDied","Data":"115c409ea3f21c0b06bc4b1e7c80ef095a4d62f4d2b43ec843516d910fc82209"}
Nov 03 23:50:05.387780 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.387550    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="115c409ea3f21c0b06bc4b1e7c80ef095a4d62f4d2b43ec843516d910fc82209"
Nov 03 23:50:05.402700 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.402679    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:50:05.516616 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.516577    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.516868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.516657    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.516938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.516891    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.516938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.516931    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.517065 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517041    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.517277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517142    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.517277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517172    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.517277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517200    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.517277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517200    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.517277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517231    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket\") pod \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\" (UID: \"54f725dc-341e-4ce2-a4dc-4385e6d8afab\") "
Nov 03 23:50:05.517564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517511    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.517842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517799    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.517946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.517878    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.518055 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.518037    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.518121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.518060    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.518121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.518079    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.518121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.518094    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.518448 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.518430    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:05.520522 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.520494    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:05.520617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.520550    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "kube-api-access-sdzv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:05.520617 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.520552    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.521717 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.521687    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token" pod "54f725dc-341e-4ce2-a4dc-4385e6d8afab" (UID: "54f725dc-341e-4ce2-a4dc-4385e6d8afab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:05.619518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.619477    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.619518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.619511    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.619518 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.619524    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.619868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.619536    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f725dc-341e-4ce2-a4dc-4385e6d8afab-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.619868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:05.619550    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzv8\" (UniqueName: \"kubernetes.io/projected/54f725dc-341e-4ce2-a4dc-4385e6d8afab-kube-api-access-sdzv8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:06.448648 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:06.447184    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"
Nov 03 23:50:06.485343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:06.485311    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"]
Nov 03 23:50:06.489787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:06.489759    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-rvntf"]
Nov 03 23:50:07.024658 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:07.024595    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" path="/var/lib/kubelet/pods/54f725dc-341e-4ce2-a4dc-4385e6d8afab/volumes"
Nov 03 23:50:09.771946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:09.771911    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"]
Nov 03 23:50:09.773249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:09.773209    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f" podUID="74f6c664-c4db-4d8d-9e0e-8cbb23d26358" containerName="payload-processing" containerID="cri-o://db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7" gracePeriod=30
Nov 03 23:50:10.292819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.292774    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:50:10.377848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.377803    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7\" (UniqueName: \"kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7\") pod \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") "
Nov 03 23:50:10.378062 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.377999    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume\") pod \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\" (UID: \"74f6c664-c4db-4d8d-9e0e-8cbb23d26358\") "
Nov 03 23:50:10.378436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.378408    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume" pod "74f6c664-c4db-4d8d-9e0e-8cbb23d26358" (UID: "74f6c664-c4db-4d8d-9e0e-8cbb23d26358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:10.378625 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.378607    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:10.380445 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.380416    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7" pod "74f6c664-c4db-4d8d-9e0e-8cbb23d26358" (UID: "74f6c664-c4db-4d8d-9e0e-8cbb23d26358"). InnerVolumeSpecName "kube-api-access-j2fc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:10.488889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.488848    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fc7\" (UniqueName: \"kubernetes.io/projected/74f6c664-c4db-4d8d-9e0e-8cbb23d26358-kube-api-access-j2fc7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:10.492289 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.492178    5074 generic.go:356] "Generic (PLEG): container finished" podID="74f6c664-c4db-4d8d-9e0e-8cbb23d26358" containerID="db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7" exitCode=0
Nov 03 23:50:10.492449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.492294    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"
Nov 03 23:50:10.492449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.492432    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f" event={"ID":"74f6c664-c4db-4d8d-9e0e-8cbb23d26358","Type":"ContainerDied","Data":"db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7"}
Nov 03 23:50:10.492568 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.492482    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f" event={"ID":"74f6c664-c4db-4d8d-9e0e-8cbb23d26358","Type":"ContainerDied","Data":"f3e8fbc6d1c6785b16335237a2190b45a3c206390283c9debc98667116dc7636"}
Nov 03 23:50:10.492630 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.492509    5074 scope.go:122] "RemoveContainer" containerID="db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7"
Nov 03 23:50:10.527789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.527672    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"]
Nov 03 23:50:10.532938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.531059    5074 scope.go:122] "RemoveContainer" containerID="db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7"
Nov 03 23:50:10.535041 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:50:10.534885    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7\": container with ID starting with db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7 not found: ID does not exist" containerID="db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7"
Nov 03 23:50:10.535041 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.534946    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7"} err="failed to get container status \"db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7\": rpc error: code = NotFound desc = could not find container \"db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7\": container with ID starting with db47f5253ae3b4496e10c19fc099515b9bfaad1836c189c11c0c23a85621dee7 not found: ID does not exist"
Nov 03 23:50:10.544838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:10.544808    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-57f4c4cc96-m995f"]
Nov 03 23:50:11.021092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:11.021059    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f6c664-c4db-4d8d-9e0e-8cbb23d26358" path="/var/lib/kubelet/pods/74f6c664-c4db-4d8d-9e0e-8cbb23d26358/volumes"
Nov 03 23:50:12.050282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:12.050235    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"]
Nov 03 23:50:12.053800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:12.053767    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0b6e31-j2rgk"]
Nov 03 23:50:13.028560 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:13.028519    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f16b6b-5302-48c0-819b-757f3e34ea20" path="/var/lib/kubelet/pods/d3f16b6b-5302-48c0-819b-757f3e34ea20/volumes"
Nov 03 23:50:28.033781 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:50:28.029799    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:50:35.095323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:35.095284    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"]
Nov 03 23:50:35.095798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:35.095496    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy" containerID="cri-o://ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af" gracePeriod=30
Nov 03 23:50:40.421383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.421353    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:50:40.430189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430161    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430246    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430265    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430304    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430344    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430409    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430546    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430545    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.430721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430642    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430681    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy\") pod \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\" (UID: \"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5\") "
Nov 03 23:50:40.430864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430758    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.430864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430784    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:40.430864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430815    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.431021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.430982    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.431472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.431376    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.431472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.431406    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.431472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.431423    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.431472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.431437    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.431472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.431452    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.433210 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.433185    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:40.433330 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.433310    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:40.433636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.433616    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "kube-api-access-xcfwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:40.434049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.434024    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy" pod "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" (UID: "0a9fd0ee-6312-4c25-a52b-05d4f0e886d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.533069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.533030    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.533069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.533063    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.533069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.533073    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfwm\" (UniqueName: \"kubernetes.io/projected/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-kube-api-access-xcfwm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.533317 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.533082    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.895520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.895478    5074 generic.go:356] "Generic (PLEG): container finished" podID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerID="ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af" exitCode=0
Nov 03 23:50:40.895721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.895695    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" event={"ID":"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5","Type":"ContainerDied","Data":"ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af"}
Nov 03 23:50:40.895833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.895766    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr" event={"ID":"0a9fd0ee-6312-4c25-a52b-05d4f0e886d5","Type":"ContainerDied","Data":"b37caa320e893f21e09f07190dfffbaa46dce1b6893322e043da0e9e064ac166"}
Nov 03 23:50:40.895833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.895786    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"
Nov 03 23:50:40.895833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.895804    5074 scope.go:122] "RemoveContainer" containerID="ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af"
Nov 03 23:50:40.911086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.911063    5074 scope.go:122] "RemoveContainer" containerID="ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af"
Nov 03 23:50:40.911413 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:50:40.911394    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af\": container with ID starting with ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af not found: ID does not exist" containerID="ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af"
Nov 03 23:50:40.911464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.911427    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af"} err="failed to get container status \"ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af\": rpc error: code = NotFound desc = could not find container \"ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af\": container with ID starting with ed176c7efd94ddd79fed7561b2d4068722b7d4401d61ec1954b6983fcaeb95af not found: ID does not exist"
Nov 03 23:50:40.929503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.929468    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"]
Nov 03 23:50:40.934103 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:40.934077    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-pbndr"]
Nov 03 23:50:41.020603 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:41.020564    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" path="/var/lib/kubelet/pods/0a9fd0ee-6312-4c25-a52b-05d4f0e886d5/volumes"
Nov 03 23:50:55.848671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.848630    5074 scope.go:122] "RemoveContainer" containerID="d57e2550bffed2d0e79845b61ad029f508427e84bd0e97c03dcd184165af12c2"
Nov 03 23:50:55.860881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.860849    5074 scope.go:122] "RemoveContainer" containerID="9a7a54e121ad15bf22d6281d695a7a7e3055cd04722e72c1da97cfc4f2376a8c"
Nov 03 23:50:55.871770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.871706    5074 scope.go:122] "RemoveContainer" containerID="0f9d6b155463477d6e63c5a5e8af9da3d0b524ed02487d468e3569b1f7a3f72b"
Nov 03 23:50:55.883466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.883434    5074 scope.go:122] "RemoveContainer" containerID="715286f7a99e0968884e57d4f3b0b58bf5ad02e14d56871eb7eb28bb3e973df1"
Nov 03 23:50:55.903117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.903080    5074 scope.go:122] "RemoveContainer" containerID="65d35adfab4193f54f8071b2dcdd7e62c25d5b5dced5d6a286578421a2760fd5"
Nov 03 23:50:55.914423 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.914396    5074 scope.go:122] "RemoveContainer" containerID="619d2752911468e0d2aa2bae748503e5c0ebb9b2d53885b7c68eac356310fa31"
Nov 03 23:50:55.926356 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.926332    5074 scope.go:122] "RemoveContainer" containerID="77d2f5a0a5db1cc8ccf7f26e7db41510cc1139f46e811038df7b3d681a963515"
Nov 03 23:50:55.938411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.938325    5074 scope.go:122] "RemoveContainer" containerID="b64fe1cb83d352e7bf791a4e45f23a2e70297667ba5c4862edd2fe0814cddcaa"
Nov 03 23:50:55.949697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.949592    5074 scope.go:122] "RemoveContainer" containerID="864391cd090f6ebc69ab6b059f59f6a36c884f28f3522e63f578b6071d390b5e"
Nov 03 23:50:55.979336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.979314    5074 scope.go:122] "RemoveContainer" containerID="98a4cef26ede6c1c4d173c60e078b32717e3063cbf162cb6eeba2b81b7e64270"
Nov 03 23:50:55.995547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:55.995523    5074 scope.go:122] "RemoveContainer" containerID="1bdfb410683574db3e93223a919d7be30d5fc491037aa9e0d123d62a83e07bb3"
Nov 03 23:50:58.023558 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:50:58.023522    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:50:58.835050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835013    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"]
Nov 03 23:50:58.835654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835638    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e14b2c07-8321-4269-9b31-53dba3a5a54b" containerName="payload-processing"
Nov 03 23:50:58.835654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835652    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14b2c07-8321-4269-9b31-53dba3a5a54b" containerName="payload-processing"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835674    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835679    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835692    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e860ce2d-1755-43cd-8b32-d41f442b620c" containerName="payload-pre-processing"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835698    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e860ce2d-1755-43cd-8b32-d41f442b620c" containerName="payload-pre-processing"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835705    5074 cpu_manager.go:414] "containerMap: removing container" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835710    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835733    5074 cpu_manager.go:414] "containerMap: removing container" podUID="74f6c664-c4db-4d8d-9e0e-8cbb23d26358" containerName="payload-processing"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835753    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f6c664-c4db-4d8d-9e0e-8cbb23d26358" containerName="payload-processing"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835847    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f6c664-c4db-4d8d-9e0e-8cbb23d26358" containerName="payload-processing"
Nov 03 23:50:58.835854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835857    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14b2c07-8321-4269-9b31-53dba3a5a54b" containerName="payload-processing"
Nov 03 23:50:58.836343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835866    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9fd0ee-6312-4c25-a52b-05d4f0e886d5" containerName="istio-proxy"
Nov 03 23:50:58.836343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835872    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e860ce2d-1755-43cd-8b32-d41f442b620c" containerName="payload-pre-processing"
Nov 03 23:50:58.836343 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.835877    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f725dc-341e-4ce2-a4dc-4385e6d8afab" containerName="istio-proxy"
Nov 03 23:50:58.840891 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.840868    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:58.848964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.848940    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c20f75\""
Nov 03 23:50:58.856904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.856883    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"]
Nov 03 23:50:58.995220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.995184    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2f9q\" (UniqueName: \"kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:58.995573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:58.995545    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:59.053580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.053478    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"]
Nov 03 23:50:59.059188 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.059161    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.091584 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.091498    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c20f75-dockercfg-2tzdv\""
Nov 03 23:50:59.099129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.098187    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"]
Nov 03 23:50:59.099293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.099218    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:59.099351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.099299    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2f9q\" (UniqueName: \"kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:59.100774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.100712    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:59.157586 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.157553    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2f9q\" (UniqueName: \"kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:59.163396 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.163363    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:50:59.205015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.200970    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.205015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.201275    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v224h\" (UniqueName: \"kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h\") pod \"payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.302260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.302227    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.302414 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.302403    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v224h\" (UniqueName: \"kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h\") pod \"payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.303439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.303381    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.327191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.325140    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v224h\" (UniqueName: \"kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h\") pod \"payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.403530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.403440    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:50:59.489940 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.489248    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"]
Nov 03 23:50:59.502997 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.500989    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27_d934e1b5-5031-4075-96f3-70728736b37f/payload-processing/0.log"
Nov 03 23:50:59.733983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:50:59.733949    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"]
Nov 03 23:50:59.734784 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:50:59.734601    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7b3e97_94ca_48bd_acb3_14746fd90ddd.slice/crio-7b2ac0d02a91f924aa60c1e3e88d3ce3b5471a06144accee66d888aa6e12d003 WatchSource:0}: Error finding container 7b2ac0d02a91f924aa60c1e3e88d3ce3b5471a06144accee66d888aa6e12d003: Status 404 returned error can't find the container with id 7b2ac0d02a91f924aa60c1e3e88d3ce3b5471a06144accee66d888aa6e12d003
Nov 03 23:51:00.128327 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.128295    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2_f209157f-4001-47b8-bf92-293790d63df4/payload-processing/0.log"
Nov 03 23:51:00.148806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.147275    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl" event={"ID":"7d7b3e97-94ca-48bd-acb3-14746fd90ddd","Type":"ContainerStarted","Data":"1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe"}
Nov 03 23:51:00.148806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.147342    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl" event={"ID":"7d7b3e97-94ca-48bd-acb3-14746fd90ddd","Type":"ContainerStarted","Data":"7b2ac0d02a91f924aa60c1e3e88d3ce3b5471a06144accee66d888aa6e12d003"}
Nov 03 23:51:00.148806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.147386    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:51:00.150861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.150008    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g" event={"ID":"e287a978-0104-48a7-a6e0-d2ede9c1e1ec","Type":"ContainerStarted","Data":"08720ca0bf39bbdd4becec7c06bca89e9bbf626bea98550def215cb9da7f1916"}
Nov 03 23:51:00.150861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.150063    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g" event={"ID":"e287a978-0104-48a7-a6e0-d2ede9c1e1ec","Type":"ContainerStarted","Data":"f5d84ba389812a2bc8532d5366a0d217391b01706ce5ba3f877a9e9cd3c21dc8"}
Nov 03 23:51:00.150861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.150106    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:51:00.173683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.173628    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl" podStartSLOduration=2.173613577 podStartE2EDuration="2.173613577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:00.170582949 +0000 UTC m=+1325.270183094" watchObservedRunningTime="2026-11-03 23:51:00.173613577 +0000 UTC m=+1325.273213722"
Nov 03 23:51:00.193662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.193588    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g" podStartSLOduration=2.193569644 podStartE2EDuration="2.193569644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:00.190868274 +0000 UTC m=+1325.290468421" watchObservedRunningTime="2026-11-03 23:51:00.193569644 +0000 UTC m=+1325.293169792"
Nov 03 23:51:00.310626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:00.310599    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2_f209157f-4001-47b8-bf92-293790d63df4/payload-processing/0.log"
Nov 03 23:51:03.264470 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.264433    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"]
Nov 03 23:51:03.269108 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.269084    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.285947 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.285809    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-beffe3-dockercfg-qccfw\""
Nov 03 23:51:03.285947 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.285876    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-beffe3\""
Nov 03 23:51:03.291331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.291302    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"]
Nov 03 23:51:03.307312 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.301590    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"]
Nov 03 23:51:03.307312 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.306972    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.313762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.311462    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"]
Nov 03 23:51:03.377493 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.377458    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.377830 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.377805    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4lj\" (UniqueName: \"kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj\") pod \"payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.488877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.488653    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.488877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.488843    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.489128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.489046    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4lj\" (UniqueName: \"kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj\") pod \"payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.489128 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.489079    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zkh\" (UniqueName: \"kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh\") pod \"payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.500719 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.496280    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.559537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.557010    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4lj\" (UniqueName: \"kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj\") pod \"payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.583387 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.583357    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:03.590003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.589978    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zkh\" (UniqueName: \"kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh\") pod \"payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.590123 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.590089    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.590840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.590813    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.637504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.637341    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zkh\" (UniqueName: \"kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh\") pod \"payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:03.874001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.873967    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"]
Nov 03 23:51:03.874463 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:51:03.874436    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b8c889_6fb1_4018_add7_42824e56c4e5.slice/crio-808fc399531306d1bea1885add9db5cf01136daa3254da258b6927e7462f7e96 WatchSource:0}: Error finding container 808fc399531306d1bea1885add9db5cf01136daa3254da258b6927e7462f7e96: Status 404 returned error can't find the container with id 808fc399531306d1bea1885add9db5cf01136daa3254da258b6927e7462f7e96
Nov 03 23:51:03.928505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:03.928481    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:04.185822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:04.185791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b" event={"ID":"d3b8c889-6fb1-4018-add7-42824e56c4e5","Type":"ContainerStarted","Data":"128d0fcd1827314ede73d886c346dc5bdafa30130445eec9354da3848440e5ec"}
Nov 03 23:51:04.186003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:04.185846    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b" event={"ID":"d3b8c889-6fb1-4018-add7-42824e56c4e5","Type":"ContainerStarted","Data":"808fc399531306d1bea1885add9db5cf01136daa3254da258b6927e7462f7e96"}
Nov 03 23:51:04.186003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:04.185930    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:04.209672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:04.209618    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b" podStartSLOduration=1.209602577 podStartE2EDuration="1.209602577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:04.205377809 +0000 UTC m=+1329.304977966" watchObservedRunningTime="2026-11-03 23:51:04.209602577 +0000 UTC m=+1329.309202722"
Nov 03 23:51:04.232042 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:04.232016    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"]
Nov 03 23:51:04.234171 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:51:04.234141    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb19a99_0fdc_49e3_b3e6_3a486bdb101f.slice/crio-3b80a03f2387165057ea1efbe0710590195130a3c4eea8145299b1be4a3c208b WatchSource:0}: Error finding container 3b80a03f2387165057ea1efbe0710590195130a3c4eea8145299b1be4a3c208b: Status 404 returned error can't find the container with id 3b80a03f2387165057ea1efbe0710590195130a3c4eea8145299b1be4a3c208b
Nov 03 23:51:05.196406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:05.196357    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" event={"ID":"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f","Type":"ContainerStarted","Data":"90ec4f35c8148d09e4bc236fe693e057a6fb9cb0e4c89993114a39f67884472c"}
Nov 03 23:51:05.196921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:05.196425    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" event={"ID":"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f","Type":"ContainerStarted","Data":"3b80a03f2387165057ea1efbe0710590195130a3c4eea8145299b1be4a3c208b"}
Nov 03 23:51:05.196921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:05.196455    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:11.161345 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:11.161313    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:51:11.161868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:11.161392    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:51:11.190167 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:11.190104    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" podStartSLOduration=8.190084056 podStartE2EDuration="8.190084056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:05.229140915 +0000 UTC m=+1330.328741060" watchObservedRunningTime="2026-11-03 23:51:11.190084056 +0000 UTC m=+1336.289684202"
Nov 03 23:51:15.203749 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:15.203713    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:51:16.210599 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:16.210574    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:51:28.024989 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:51:28.024947    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:51:30.269512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:30.269474    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"]
Nov 03 23:51:30.270171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:30.269812    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy" containerID="cri-o://f3629049f7bb80e1166a03027682cf80e70500f6e9280f889384b7bd95f24c9e" gracePeriod=30
Nov 03 23:51:35.031208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.031165    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"]
Nov 03 23:51:35.035060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.035031    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875d96be-tsv8r"]
Nov 03 23:51:35.552363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.552282    5074 generic.go:356] "Generic (PLEG): container finished" podID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerID="f3629049f7bb80e1166a03027682cf80e70500f6e9280f889384b7bd95f24c9e" exitCode=0
Nov 03 23:51:35.552363 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.552346    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" event={"ID":"2e3abbca-9c07-4b0e-a9d0-0544fedffc27","Type":"ContainerDied","Data":"f3629049f7bb80e1166a03027682cf80e70500f6e9280f889384b7bd95f24c9e"}
Nov 03 23:51:35.593503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.593473    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:51:35.661109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661081    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661127    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661155    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661240    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661274    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661324    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661399    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661465    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661583    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token\") pod \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\" (UID: \"2e3abbca-9c07-4b0e-a9d0-0544fedffc27\") "
Nov 03 23:51:35.661644 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661584    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:35.661876 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.661851    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:35.662097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662071    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:35.662259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662224    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:35.662368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662347    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.662443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662376    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.662443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662392    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.662443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662406    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.662580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.662566    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:35.664789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.664710    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:35.665085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.665034    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:35.665184 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.665110    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:51:35.665601 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.665564    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn" pod "2e3abbca-9c07-4b0e-a9d0-0544fedffc27" (UID: "2e3abbca-9c07-4b0e-a9d0-0544fedffc27"). InnerVolumeSpecName "kube-api-access-xpdzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:35.764631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.764581    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.764631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.764624    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.764941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.764641    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpdzn\" (UniqueName: \"kubernetes.io/projected/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-kube-api-access-xpdzn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.764941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.764658    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.764941 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:35.764672    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3abbca-9c07-4b0e-a9d0-0544fedffc27-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:36.561786 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:36.561721    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"
Nov 03 23:51:36.562292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:36.561783    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz" event={"ID":"2e3abbca-9c07-4b0e-a9d0-0544fedffc27","Type":"ContainerDied","Data":"ef80489c36445a03e89d00a95ad7c2d8cc60e22fed327a9535a47f832fc9657f"}
Nov 03 23:51:36.562292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:36.561859    5074 scope.go:122] "RemoveContainer" containerID="f3629049f7bb80e1166a03027682cf80e70500f6e9280f889384b7bd95f24c9e"
Nov 03 23:51:36.589279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:36.589251    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"]
Nov 03 23:51:36.596715 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:36.596690    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-lgzfz"]
Nov 03 23:51:37.021797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:37.021766    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" path="/var/lib/kubelet/pods/2e3abbca-9c07-4b0e-a9d0-0544fedffc27/volumes"
Nov 03 23:51:37.022377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:37.022346    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7059414f-1b8f-4233-9cff-dbf154466754" path="/var/lib/kubelet/pods/7059414f-1b8f-4233-9cff-dbf154466754/volumes"
Nov 03 23:51:43.547185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.547143    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"]
Nov 03 23:51:43.548170 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.548146    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy"
Nov 03 23:51:43.548231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.548174    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy"
Nov 03 23:51:43.548430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.548416    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3abbca-9c07-4b0e-a9d0-0544fedffc27" containerName="istio-proxy"
Nov 03 23:51:43.552753 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.552717    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.562826 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.562790    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"]
Nov 03 23:51:43.569179 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.566845    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:51:43.574017 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.573989    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:51:43.636547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.636034    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.636547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.636164    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbht9\" (UniqueName: \"kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.636547 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.636211    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.737349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.737309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.737555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.737455    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbht9\" (UniqueName: \"kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.737555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.737503    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.738112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.738079    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.739914 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.739892    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.750822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.750795    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbht9\" (UniqueName: \"kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9\") pod \"maas-api-revoke-sub-66d50c6f-hgbfl\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:43.881392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:43.881314    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:44.220539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:44.214962    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"]
Nov 03 23:51:44.655262 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:44.655222    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl" event={"ID":"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b","Type":"ContainerStarted","Data":"f8af3ec3d22756fcbcab64f17e70145e2d48581246774fed013956e0697584a3"}
Nov 03 23:51:44.655706 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:44.655329    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl" event={"ID":"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b","Type":"ContainerStarted","Data":"2524deba800dca41a64a81a18d50ed25e05d3e0f73865fc0afd7d81019831033"}
Nov 03 23:51:44.678551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:44.678498    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl" podStartSLOduration=1.6784842150000001 podStartE2EDuration="1.678484215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:44.674382402 +0000 UTC m=+1369.773982549" watchObservedRunningTime="2026-11-03 23:51:44.678484215 +0000 UTC m=+1369.778084358"
Nov 03 23:51:45.665898 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:45.665812    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" containerID="f8af3ec3d22756fcbcab64f17e70145e2d48581246774fed013956e0697584a3" exitCode=0
Nov 03 23:51:45.666287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:45.665888    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl" event={"ID":"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b","Type":"ContainerDied","Data":"f8af3ec3d22756fcbcab64f17e70145e2d48581246774fed013956e0697584a3"}
Nov 03 23:51:46.887927 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.887897    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:46.888274 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.888247    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9\" (UniqueName: \"kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9\") pod \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") "
Nov 03 23:51:46.888387 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.888370    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca\") pod \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") "
Nov 03 23:51:46.888458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.888443    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca\") pod \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\" (UID: \"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b\") "
Nov 03 23:51:46.888917 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.888887    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca" pod "a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" (UID: "a3d1cd17-a7ef-4111-92b3-9344f9d4c26b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:46.889053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.889038    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:46.890649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.890624    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9" pod "a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" (UID: "a3d1cd17-a7ef-4111-92b3-9344f9d4c26b"). InnerVolumeSpecName "kube-api-access-qbht9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:46.890862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.890840    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca" pod "a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" (UID: "a3d1cd17-a7ef-4111-92b3-9344f9d4c26b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:51:46.990755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.990710    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbht9\" (UniqueName: \"kubernetes.io/projected/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-kube-api-access-qbht9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:46.990755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:46.990761    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:47.684255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:47.684225    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"
Nov 03 23:51:47.684255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:47.684249    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl" event={"ID":"a3d1cd17-a7ef-4111-92b3-9344f9d4c26b","Type":"ContainerDied","Data":"2524deba800dca41a64a81a18d50ed25e05d3e0f73865fc0afd7d81019831033"}
Nov 03 23:51:47.684877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:47.684299    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2524deba800dca41a64a81a18d50ed25e05d3e0f73865fc0afd7d81019831033"
Nov 03 23:51:53.757875 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.757845    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"]
Nov 03 23:51:53.758555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.758539    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" containerName="revoke-keys"
Nov 03 23:51:53.758602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.758556    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" containerName="revoke-keys"
Nov 03 23:51:53.758671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.758662    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" containerName="revoke-keys"
Nov 03 23:51:53.762481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.762464    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:53.772713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.772684    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"]
Nov 03 23:51:53.799301 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.799262    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nll8q\" (UniqueName: \"kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:53.799479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.799417    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:53.852078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.851970    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5f5cd2\""
Nov 03 23:51:53.852391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.852371    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5f5cd2-dockercfg-ltjvz\""
Nov 03 23:51:53.904762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.901332    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nll8q\" (UniqueName: \"kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:53.904762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.901461    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:53.921762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.911560    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:53.956762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:53.955843    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nll8q\" (UniqueName: \"kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:54.079788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:54.079691    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:54.378001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:54.377961    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"]
Nov 03 23:51:54.378586 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:51:54.378558    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59ba856_3f73_4ff7_93e3_ef3b64b37548.slice/crio-a91d10038f020cc5616588d13c6e403b5fc491697a9cb635b770053979c84771 WatchSource:0}: Error finding container a91d10038f020cc5616588d13c6e403b5fc491697a9cb635b770053979c84771: Status 404 returned error can't find the container with id a91d10038f020cc5616588d13c6e403b5fc491697a9cb635b770053979c84771
Nov 03 23:51:54.756419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:54.756388    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2" event={"ID":"c59ba856-3f73-4ff7-93e3-ef3b64b37548","Type":"ContainerStarted","Data":"6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88"}
Nov 03 23:51:54.756612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:54.756439    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2" event={"ID":"c59ba856-3f73-4ff7-93e3-ef3b64b37548","Type":"ContainerStarted","Data":"a91d10038f020cc5616588d13c6e403b5fc491697a9cb635b770053979c84771"}
Nov 03 23:51:54.756612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:54.756518    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:51:54.780795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:54.780721    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2" podStartSLOduration=1.780708782 podStartE2EDuration="1.780708782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:54.777246943 +0000 UTC m=+1379.876847087" watchObservedRunningTime="2026-11-03 23:51:54.780708782 +0000 UTC m=+1379.880308926"
Nov 03 23:51:56.096504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:51:56.096461    5074 scope.go:122] "RemoveContainer" containerID="a02eb76007ac116e8eb25c654ac263c162d2e1e03e32eca1b1fdbdc7d437b79a"
Nov 03 23:51:58.029770 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:51:58.027556    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:52:04.376590 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.376557    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"]
Nov 03 23:52:04.377395 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.377360    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27" podUID="d934e1b5-5031-4075-96f3-70728736b37f" containerName="payload-processing" containerID="cri-o://1be397144fedb6f1c3ec49109c2a04d923a19c25d74373516a47513b61c0fbee" gracePeriod=30
Nov 03 23:52:04.387142 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.387119    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"]
Nov 03 23:52:04.387505 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.387466    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf" podUID="8e409d6a-dd66-40db-bec0-8983076a8d4a" containerName="payload-pre-processing" containerID="cri-o://ade18943291bccb12f4b67278cf52c52a7b24916a22859cdfa4561e67bba9875" gracePeriod=30
Nov 03 23:52:04.849759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.849659    5074 generic.go:356] "Generic (PLEG): container finished" podID="d934e1b5-5031-4075-96f3-70728736b37f" containerID="1be397144fedb6f1c3ec49109c2a04d923a19c25d74373516a47513b61c0fbee" exitCode=0
Nov 03 23:52:04.852779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.849859    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27" event={"ID":"d934e1b5-5031-4075-96f3-70728736b37f","Type":"ContainerDied","Data":"1be397144fedb6f1c3ec49109c2a04d923a19c25d74373516a47513b61c0fbee"}
Nov 03 23:52:04.852779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.852325    5074 generic.go:356] "Generic (PLEG): container finished" podID="8e409d6a-dd66-40db-bec0-8983076a8d4a" containerID="ade18943291bccb12f4b67278cf52c52a7b24916a22859cdfa4561e67bba9875" exitCode=0
Nov 03 23:52:04.852779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.852484    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf" event={"ID":"8e409d6a-dd66-40db-bec0-8983076a8d4a","Type":"ContainerDied","Data":"ade18943291bccb12f4b67278cf52c52a7b24916a22859cdfa4561e67bba9875"}
Nov 03 23:52:05.002781 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:04.994769    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:52:05.075880 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.075088    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:52:05.132446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.132392    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh\" (UniqueName: \"kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh\") pod \"8e409d6a-dd66-40db-bec0-8983076a8d4a\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") "
Nov 03 23:52:05.132640 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.132514    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume\") pod \"8e409d6a-dd66-40db-bec0-8983076a8d4a\" (UID: \"8e409d6a-dd66-40db-bec0-8983076a8d4a\") "
Nov 03 23:52:05.133996 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.133966    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume" pod "8e409d6a-dd66-40db-bec0-8983076a8d4a" (UID: "8e409d6a-dd66-40db-bec0-8983076a8d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:05.135550 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.135521    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh" pod "8e409d6a-dd66-40db-bec0-8983076a8d4a" (UID: "8e409d6a-dd66-40db-bec0-8983076a8d4a"). InnerVolumeSpecName "kube-api-access-2wfnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:05.234375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.234342    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq\" (UniqueName: \"kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq\") pod \"d934e1b5-5031-4075-96f3-70728736b37f\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") "
Nov 03 23:52:05.234566 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.234522    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume\") pod \"d934e1b5-5031-4075-96f3-70728736b37f\" (UID: \"d934e1b5-5031-4075-96f3-70728736b37f\") "
Nov 03 23:52:05.234997 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.234961    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume" pod "d934e1b5-5031-4075-96f3-70728736b37f" (UID: "d934e1b5-5031-4075-96f3-70728736b37f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:05.235190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.235171    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d934e1b5-5031-4075-96f3-70728736b37f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:05.235260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.235192    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfnh\" (UniqueName: \"kubernetes.io/projected/8e409d6a-dd66-40db-bec0-8983076a8d4a-kube-api-access-2wfnh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:05.235260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.235203    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e409d6a-dd66-40db-bec0-8983076a8d4a-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:05.236479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.236449    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq" pod "d934e1b5-5031-4075-96f3-70728736b37f" (UID: "d934e1b5-5031-4075-96f3-70728736b37f"). InnerVolumeSpecName "kube-api-access-zzwbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:05.336248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.336166    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwbq\" (UniqueName: \"kubernetes.io/projected/d934e1b5-5031-4075-96f3-70728736b37f-kube-api-access-zzwbq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:05.766364 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.766334    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:52:05.872185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.872155    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"
Nov 03 23:52:05.872432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.872184    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf" event={"ID":"8e409d6a-dd66-40db-bec0-8983076a8d4a","Type":"ContainerDied","Data":"04f6035f80c30b011f218a128ba407e39e58bd7bb4570e074601c2b29b3afc5f"}
Nov 03 23:52:05.872432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.872244    5074 scope.go:122] "RemoveContainer" containerID="ade18943291bccb12f4b67278cf52c52a7b24916a22859cdfa4561e67bba9875"
Nov 03 23:52:05.874572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.874508    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27" event={"ID":"d934e1b5-5031-4075-96f3-70728736b37f","Type":"ContainerDied","Data":"aebc5e8c3c8265b0277072b953b94def52177dd9803ee0b60194b2c4bda5085d"}
Nov 03 23:52:05.874675 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.874656    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"
Nov 03 23:52:05.891680 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.891602    5074 scope.go:122] "RemoveContainer" containerID="1be397144fedb6f1c3ec49109c2a04d923a19c25d74373516a47513b61c0fbee"
Nov 03 23:52:05.914138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.914109    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"]
Nov 03 23:52:05.918844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.918054    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-85b7b9b5c5-bxhsf"]
Nov 03 23:52:05.927278 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.927253    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"]
Nov 03 23:52:05.931538 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:05.931504    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-7c4448c4db-pfx27"]
Nov 03 23:52:07.021713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:07.021671    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e409d6a-dd66-40db-bec0-8983076a8d4a" path="/var/lib/kubelet/pods/8e409d6a-dd66-40db-bec0-8983076a8d4a/volumes"
Nov 03 23:52:07.022361 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:07.022338    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d934e1b5-5031-4075-96f3-70728736b37f" path="/var/lib/kubelet/pods/d934e1b5-5031-4075-96f3-70728736b37f/volumes"
Nov 03 23:52:15.022706 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.022662    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"]
Nov 03 23:52:15.023673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.023648    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8e409d6a-dd66-40db-bec0-8983076a8d4a" containerName="payload-pre-processing"
Nov 03 23:52:15.023816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.023674    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e409d6a-dd66-40db-bec0-8983076a8d4a" containerName="payload-pre-processing"
Nov 03 23:52:15.023816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.023705    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d934e1b5-5031-4075-96f3-70728736b37f" containerName="payload-processing"
Nov 03 23:52:15.023816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.023714    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d934e1b5-5031-4075-96f3-70728736b37f" containerName="payload-processing"
Nov 03 23:52:15.023989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.023937    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e409d6a-dd66-40db-bec0-8983076a8d4a" containerName="payload-pre-processing"
Nov 03 23:52:15.023989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.023953    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d934e1b5-5031-4075-96f3-70728736b37f" containerName="payload-processing"
Nov 03 23:52:15.027668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.027644    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.032192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.032166    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:52:15.033157 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.033134    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:52:15.035190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.035169    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"]
Nov 03 23:52:15.091221 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.091182    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7l2\" (UniqueName: \"kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.091370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.091246    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.091370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.091350    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.193315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.193284    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7l2\" (UniqueName: \"kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.193480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.193335    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.193480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.193392    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.194085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.194059    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.196171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.196142    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.206803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.206771    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7l2\" (UniqueName: \"kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2\") pod \"maas-api-revoke-sub-89d513be-6pqpf\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.348203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.348108    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:15.635813 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:52:15.635730    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe83b3f_4a56_44e9_a168_51a24f031d52.slice/crio-086d3dadb8dc5e96f7f97df68e86c35b344d57e35f8a69983ba8b909179c99b1 WatchSource:0}: Error finding container 086d3dadb8dc5e96f7f97df68e86c35b344d57e35f8a69983ba8b909179c99b1: Status 404 returned error can't find the container with id 086d3dadb8dc5e96f7f97df68e86c35b344d57e35f8a69983ba8b909179c99b1
Nov 03 23:52:15.636325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.636294    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"]
Nov 03 23:52:15.993993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.993953    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf" event={"ID":"ffe83b3f-4a56-44e9-a168-51a24f031d52","Type":"ContainerStarted","Data":"4b48a20adc7f251677e48f1984b9915dfc768e7e36ad9eea3539237244d65889"}
Nov 03 23:52:15.994197 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:15.994022    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf" event={"ID":"ffe83b3f-4a56-44e9-a168-51a24f031d52","Type":"ContainerStarted","Data":"086d3dadb8dc5e96f7f97df68e86c35b344d57e35f8a69983ba8b909179c99b1"}
Nov 03 23:52:16.018369 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:16.018319    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf" podStartSLOduration=1.018305671 podStartE2EDuration="1.018305671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:16.016029318 +0000 UTC m=+1401.115629463" watchObservedRunningTime="2026-11-03 23:52:16.018305671 +0000 UTC m=+1401.117905815"
Nov 03 23:52:16.036374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:16.036337    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"]
Nov 03 23:52:16.036984 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:16.036428    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cb4681-k9nnb"]
Nov 03 23:52:17.005994 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:17.005950    5074 generic.go:356] "Generic (PLEG): container finished" podID="ffe83b3f-4a56-44e9-a168-51a24f031d52" containerID="4b48a20adc7f251677e48f1984b9915dfc768e7e36ad9eea3539237244d65889" exitCode=0
Nov 03 23:52:17.006228 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:17.006048    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf" event={"ID":"ffe83b3f-4a56-44e9-a168-51a24f031d52","Type":"ContainerDied","Data":"4b48a20adc7f251677e48f1984b9915dfc768e7e36ad9eea3539237244d65889"}
Nov 03 23:52:17.020609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:17.020567    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d16fe35-ca87-45d9-aad8-be4149b6b2bd" path="/var/lib/kubelet/pods/1d16fe35-ca87-45d9-aad8-be4149b6b2bd/volumes"
Nov 03 23:52:18.223672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.223476    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:18.348267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.348180    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca\") pod \"ffe83b3f-4a56-44e9-a168-51a24f031d52\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") "
Nov 03 23:52:18.348267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.348221    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2\" (UniqueName: \"kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2\") pod \"ffe83b3f-4a56-44e9-a168-51a24f031d52\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") "
Nov 03 23:52:18.348267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.348259    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca\") pod \"ffe83b3f-4a56-44e9-a168-51a24f031d52\" (UID: \"ffe83b3f-4a56-44e9-a168-51a24f031d52\") "
Nov 03 23:52:18.348599 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.348570    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca" pod "ffe83b3f-4a56-44e9-a168-51a24f031d52" (UID: "ffe83b3f-4a56-44e9-a168-51a24f031d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:18.348687 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.348656    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:18.350716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.350687    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2" pod "ffe83b3f-4a56-44e9-a168-51a24f031d52" (UID: "ffe83b3f-4a56-44e9-a168-51a24f031d52"). InnerVolumeSpecName "kube-api-access-2b7l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:18.350880 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.350858    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca" pod "ffe83b3f-4a56-44e9-a168-51a24f031d52" (UID: "ffe83b3f-4a56-44e9-a168-51a24f031d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:18.450538 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.450501    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7l2\" (UniqueName: \"kubernetes.io/projected/ffe83b3f-4a56-44e9-a168-51a24f031d52-kube-api-access-2b7l2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:18.450538 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:18.450535    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe83b3f-4a56-44e9-a168-51a24f031d52-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:19.039583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:19.039545    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf" event={"ID":"ffe83b3f-4a56-44e9-a168-51a24f031d52","Type":"ContainerDied","Data":"086d3dadb8dc5e96f7f97df68e86c35b344d57e35f8a69983ba8b909179c99b1"}
Nov 03 23:52:19.039583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:19.039607    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="086d3dadb8dc5e96f7f97df68e86c35b344d57e35f8a69983ba8b909179c99b1"
Nov 03 23:52:19.039993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:19.039680    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"
Nov 03 23:52:20.348222 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:20.348182    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"]
Nov 03 23:52:20.359135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:20.358213    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"]
Nov 03 23:52:25.236514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:25.236475    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"]
Nov 03 23:52:25.237224 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:25.236684    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl" podUID="7d7b3e97-94ca-48bd-acb3-14746fd90ddd" containerName="payload-processing" containerID="cri-o://1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe" gracePeriod=30
Nov 03 23:52:25.305899 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:25.305858    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"]
Nov 03 23:52:25.306186 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:25.306132    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g" podUID="e287a978-0104-48a7-a6e0-d2ede9c1e1ec" containerName="payload-pre-processing" containerID="cri-o://08720ca0bf39bbdd4becec7c06bca89e9bbf626bea98550def215cb9da7f1916" gracePeriod=30
Nov 03 23:52:25.356880 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:25.351954    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b" podUID="d3b8c889-6fb1-4018-add7-42824e56c4e5" containerName="payload-processing" containerID="cri-o://128d0fcd1827314ede73d886c346dc5bdafa30130445eec9354da3848440e5ec" gracePeriod=25
Nov 03 23:52:25.361069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:25.358602    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" podUID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" containerName="payload-pre-processing" containerID="cri-o://90ec4f35c8148d09e4bc236fe693e057a6fb9cb0e4c89993114a39f67884472c" gracePeriod=25
Nov 03 23:52:26.072964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.072934    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:52:26.237376 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.237336    5074 generic.go:356] "Generic (PLEG): container finished" podID="e287a978-0104-48a7-a6e0-d2ede9c1e1ec" containerID="08720ca0bf39bbdd4becec7c06bca89e9bbf626bea98550def215cb9da7f1916" exitCode=0
Nov 03 23:52:26.238099 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.238069    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g" event={"ID":"e287a978-0104-48a7-a6e0-d2ede9c1e1ec","Type":"ContainerDied","Data":"08720ca0bf39bbdd4becec7c06bca89e9bbf626bea98550def215cb9da7f1916"}
Nov 03 23:52:26.240615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.240578    5074 generic.go:356] "Generic (PLEG): container finished" podID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" containerID="90ec4f35c8148d09e4bc236fe693e057a6fb9cb0e4c89993114a39f67884472c" exitCode=0
Nov 03 23:52:26.240901 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.240870    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" event={"ID":"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f","Type":"ContainerDied","Data":"90ec4f35c8148d09e4bc236fe693e057a6fb9cb0e4c89993114a39f67884472c"}
Nov 03 23:52:26.242879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.242856    5074 generic.go:356] "Generic (PLEG): container finished" podID="7d7b3e97-94ca-48bd-acb3-14746fd90ddd" containerID="1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe" exitCode=0
Nov 03 23:52:26.243271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.243253    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl" event={"ID":"7d7b3e97-94ca-48bd-acb3-14746fd90ddd","Type":"ContainerDied","Data":"1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe"}
Nov 03 23:52:26.243436 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.243411    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl" event={"ID":"7d7b3e97-94ca-48bd-acb3-14746fd90ddd","Type":"ContainerDied","Data":"7b2ac0d02a91f924aa60c1e3e88d3ce3b5471a06144accee66d888aa6e12d003"}
Nov 03 23:52:26.243580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.243566    5074 scope.go:122] "RemoveContainer" containerID="1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe"
Nov 03 23:52:26.243906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.243884    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"
Nov 03 23:52:26.250618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.250594    5074 generic.go:356] "Generic (PLEG): container finished" podID="d3b8c889-6fb1-4018-add7-42824e56c4e5" containerID="128d0fcd1827314ede73d886c346dc5bdafa30130445eec9354da3848440e5ec" exitCode=0
Nov 03 23:52:26.250867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.250833    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b" event={"ID":"d3b8c889-6fb1-4018-add7-42824e56c4e5","Type":"ContainerDied","Data":"128d0fcd1827314ede73d886c346dc5bdafa30130445eec9354da3848440e5ec"}
Nov 03 23:52:26.259164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.259139    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume\") pod \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") "
Nov 03 23:52:26.269398 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.259661    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h\" (UniqueName: \"kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h\") pod \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\" (UID: \"7d7b3e97-94ca-48bd-acb3-14746fd90ddd\") "
Nov 03 23:52:26.310861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.288567    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume" pod "7d7b3e97-94ca-48bd-acb3-14746fd90ddd" (UID: "7d7b3e97-94ca-48bd-acb3-14746fd90ddd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:26.310861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.289637    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.310861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.295172    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h" pod "7d7b3e97-94ca-48bd-acb3-14746fd90ddd" (UID: "7d7b3e97-94ca-48bd-acb3-14746fd90ddd"). InnerVolumeSpecName "kube-api-access-v224h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:26.310861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.301973    5074 scope.go:122] "RemoveContainer" containerID="1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe"
Nov 03 23:52:26.310861 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:52:26.302421    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe\": container with ID starting with 1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe not found: ID does not exist" containerID="1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe"
Nov 03 23:52:26.310861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.302455    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe"} err="failed to get container status \"1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe\": rpc error: code = NotFound desc = could not find container \"1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe\": container with ID starting with 1e59f50209bacfb00e0798752bad676e2c7220665ae15bea416357ea89a880fe not found: ID does not exist"
Nov 03 23:52:26.324014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.323644    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:52:26.368076 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.368044    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:52:26.372181 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.372157    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:52:26.390618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.390584    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh\" (UniqueName: \"kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh\") pod \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") "
Nov 03 23:52:26.391041 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.391020    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume\") pod \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\" (UID: \"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f\") "
Nov 03 23:52:26.391624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.391593    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v224h\" (UniqueName: \"kubernetes.io/projected/7d7b3e97-94ca-48bd-acb3-14746fd90ddd-kube-api-access-v224h\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.392157 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.392127    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume" pod "5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" (UID: "5bb19a99-0fdc-49e3-b3e6-3a486bdb101f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:26.460454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.460325    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh" pod "5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" (UID: "5bb19a99-0fdc-49e3-b3e6-3a486bdb101f"). InnerVolumeSpecName "kube-api-access-b4zkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:26.509563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.509525    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume\") pod \"d3b8c889-6fb1-4018-add7-42824e56c4e5\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") "
Nov 03 23:52:26.509855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.509833    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj\" (UniqueName: \"kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj\") pod \"d3b8c889-6fb1-4018-add7-42824e56c4e5\" (UID: \"d3b8c889-6fb1-4018-add7-42824e56c4e5\") "
Nov 03 23:52:26.510240 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.510220    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume\") pod \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") "
Nov 03 23:52:26.510401 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.510387    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q\" (UniqueName: \"kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q\") pod \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\" (UID: \"e287a978-0104-48a7-a6e0-d2ede9c1e1ec\") "
Nov 03 23:52:26.510614 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.510562    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume" pod "e287a978-0104-48a7-a6e0-d2ede9c1e1ec" (UID: "e287a978-0104-48a7-a6e0-d2ede9c1e1ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:26.511149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.511117    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.511246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.511157    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4zkh\" (UniqueName: \"kubernetes.io/projected/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-kube-api-access-b4zkh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.511246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.511173    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.511454 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.511430    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume" pod "d3b8c889-6fb1-4018-add7-42824e56c4e5" (UID: "d3b8c889-6fb1-4018-add7-42824e56c4e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:26.514719 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.514686    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj" pod "d3b8c889-6fb1-4018-add7-42824e56c4e5" (UID: "d3b8c889-6fb1-4018-add7-42824e56c4e5"). InnerVolumeSpecName "kube-api-access-mp4lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:26.523983 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.523925    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q" pod "e287a978-0104-48a7-a6e0-d2ede9c1e1ec" (UID: "e287a978-0104-48a7-a6e0-d2ede9c1e1ec"). InnerVolumeSpecName "kube-api-access-p2f9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:26.595076 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.595045    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"]
Nov 03 23:52:26.601557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.601458    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-5469dcc5dd-x8wrl"]
Nov 03 23:52:26.613006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.612974    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2f9q\" (UniqueName: \"kubernetes.io/projected/e287a978-0104-48a7-a6e0-d2ede9c1e1ec-kube-api-access-p2f9q\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.613006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.613002    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b8c889-6fb1-4018-add7-42824e56c4e5-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.613218 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:26.613013    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4lj\" (UniqueName: \"kubernetes.io/projected/d3b8c889-6fb1-4018-add7-42824e56c4e5-kube-api-access-mp4lj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:27.037711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.037672    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7b3e97-94ca-48bd-acb3-14746fd90ddd" path="/var/lib/kubelet/pods/7d7b3e97-94ca-48bd-acb3-14746fd90ddd/volumes"
Nov 03 23:52:27.229616 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.229567    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" podUID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.147:50052\" within 1s: context deadline exceeded"
Nov 03 23:52:27.229949 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:52:27.229704    5074 logging.go:55] [core] [Channel #1229 SubChannel #1230]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.147:50052", ServerName: "10.132.0.147:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.147:50052: operation was canceled"
Nov 03 23:52:27.265013 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.264972    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b" event={"ID":"d3b8c889-6fb1-4018-add7-42824e56c4e5","Type":"ContainerDied","Data":"808fc399531306d1bea1885add9db5cf01136daa3254da258b6927e7462f7e96"}
Nov 03 23:52:27.265553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.265051    5074 scope.go:122] "RemoveContainer" containerID="128d0fcd1827314ede73d886c346dc5bdafa30130445eec9354da3848440e5ec"
Nov 03 23:52:27.265553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.265182    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"
Nov 03 23:52:27.271319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.271292    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g" event={"ID":"e287a978-0104-48a7-a6e0-d2ede9c1e1ec","Type":"ContainerDied","Data":"f5d84ba389812a2bc8532d5366a0d217391b01706ce5ba3f877a9e9cd3c21dc8"}
Nov 03 23:52:27.271581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.271566    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"
Nov 03 23:52:27.275597 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.275575    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq" event={"ID":"5bb19a99-0fdc-49e3-b3e6-3a486bdb101f","Type":"ContainerDied","Data":"3b80a03f2387165057ea1efbe0710590195130a3c4eea8145299b1be4a3c208b"}
Nov 03 23:52:27.275838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.275821    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"
Nov 03 23:52:27.332791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.317574    5074 scope.go:122] "RemoveContainer" containerID="08720ca0bf39bbdd4becec7c06bca89e9bbf626bea98550def215cb9da7f1916"
Nov 03 23:52:27.350425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.350025    5074 scope.go:122] "RemoveContainer" containerID="90ec4f35c8148d09e4bc236fe693e057a6fb9cb0e4c89993114a39f67884472c"
Nov 03 23:52:27.363779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.358175    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"]
Nov 03 23:52:27.393770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.390984    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-beffe3-5b6f89cff6-84q2b"]
Nov 03 23:52:27.407131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.407075    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"]
Nov 03 23:52:27.413293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.413236    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-5c576b7d47-9qr5g"]
Nov 03 23:52:27.418810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.418765    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"]
Nov 03 23:52:27.424537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:27.424515    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-beffe3-74bbdd94c9-b4zbq"]
Nov 03 23:52:28.072488 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:52:28.072445    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:52:29.022074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:29.022043    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" path="/var/lib/kubelet/pods/5bb19a99-0fdc-49e3-b3e6-3a486bdb101f/volumes"
Nov 03 23:52:29.022506 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:29.022492    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b8c889-6fb1-4018-add7-42824e56c4e5" path="/var/lib/kubelet/pods/d3b8c889-6fb1-4018-add7-42824e56c4e5/volumes"
Nov 03 23:52:29.022872 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:29.022860    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e287a978-0104-48a7-a6e0-d2ede9c1e1ec" path="/var/lib/kubelet/pods/e287a978-0104-48a7-a6e0-d2ede9c1e1ec/volumes"
Nov 03 23:52:31.960733 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.960699    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"]
Nov 03 23:52:31.961347 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961331    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" containerName="payload-pre-processing"
Nov 03 23:52:31.961392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961347    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" containerName="payload-pre-processing"
Nov 03 23:52:31.961392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961374    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d3b8c889-6fb1-4018-add7-42824e56c4e5" containerName="payload-processing"
Nov 03 23:52:31.961392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961382    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b8c889-6fb1-4018-add7-42824e56c4e5" containerName="payload-processing"
Nov 03 23:52:31.961483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961395    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7d7b3e97-94ca-48bd-acb3-14746fd90ddd" containerName="payload-processing"
Nov 03 23:52:31.961483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961400    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7b3e97-94ca-48bd-acb3-14746fd90ddd" containerName="payload-processing"
Nov 03 23:52:31.961483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961410    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e287a978-0104-48a7-a6e0-d2ede9c1e1ec" containerName="payload-pre-processing"
Nov 03 23:52:31.961483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961415    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e287a978-0104-48a7-a6e0-d2ede9c1e1ec" containerName="payload-pre-processing"
Nov 03 23:52:31.961483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961435    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffe83b3f-4a56-44e9-a168-51a24f031d52" containerName="revoke-keys"
Nov 03 23:52:31.961483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961440    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe83b3f-4a56-44e9-a168-51a24f031d52" containerName="revoke-keys"
Nov 03 23:52:31.961662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961527    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe83b3f-4a56-44e9-a168-51a24f031d52" containerName="revoke-keys"
Nov 03 23:52:31.961662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961541    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb19a99-0fdc-49e3-b3e6-3a486bdb101f" containerName="payload-pre-processing"
Nov 03 23:52:31.961662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961550    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b8c889-6fb1-4018-add7-42824e56c4e5" containerName="payload-processing"
Nov 03 23:52:31.961662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961555    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7b3e97-94ca-48bd-acb3-14746fd90ddd" containerName="payload-processing"
Nov 03 23:52:31.961662 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.961561    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e287a978-0104-48a7-a6e0-d2ede9c1e1ec" containerName="payload-pre-processing"
Nov 03 23:52:31.966194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.966174    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:31.973378 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.973357    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:52:31.979352 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.979327    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"]
Nov 03 23:52:31.985487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:31.985457    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:52:32.017602 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.017569    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.019107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.019079    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.019487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.019467    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glklx\" (UniqueName: \"kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.120536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.120502    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.120800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.120587    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.120800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.120633    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glklx\" (UniqueName: \"kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.121410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.121376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.123220 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.123193    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.134752 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.134719    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glklx\" (UniqueName: \"kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx\") pod \"maas-api-revoke-sub-582ae5d1-lfv5v\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.293632 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.293547    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:32.595919 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:52:32.595878    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baf683e_b684_479c_9a75_7e405002c4a2.slice/crio-f4ae92e36446e65b22031651c335571bec5bcaa6296cb5dbf7781fe2a71ca2e0 WatchSource:0}: Error finding container f4ae92e36446e65b22031651c335571bec5bcaa6296cb5dbf7781fe2a71ca2e0: Status 404 returned error can't find the container with id f4ae92e36446e65b22031651c335571bec5bcaa6296cb5dbf7781fe2a71ca2e0
Nov 03 23:52:32.596391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:32.596303    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"]
Nov 03 23:52:33.343143 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:33.343110    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v" event={"ID":"8baf683e-b684-479c-9a75-7e405002c4a2","Type":"ContainerStarted","Data":"cee23b77916d7536b011bca3df2ea62018dba1ffef834993d1e38549648d6941"}
Nov 03 23:52:33.343554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:33.343169    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v" event={"ID":"8baf683e-b684-479c-9a75-7e405002c4a2","Type":"ContainerStarted","Data":"f4ae92e36446e65b22031651c335571bec5bcaa6296cb5dbf7781fe2a71ca2e0"}
Nov 03 23:52:33.368245 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:33.368180    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v" podStartSLOduration=2.368161473 podStartE2EDuration="2.368161473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:33.363417965 +0000 UTC m=+1418.463018109" watchObservedRunningTime="2026-11-03 23:52:33.368161473 +0000 UTC m=+1418.467761618"
Nov 03 23:52:34.354281 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:34.354240    5074 generic.go:356] "Generic (PLEG): container finished" podID="8baf683e-b684-479c-9a75-7e405002c4a2" containerID="cee23b77916d7536b011bca3df2ea62018dba1ffef834993d1e38549648d6941" exitCode=0
Nov 03 23:52:34.354779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:34.354333    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v" event={"ID":"8baf683e-b684-479c-9a75-7e405002c4a2","Type":"ContainerDied","Data":"cee23b77916d7536b011bca3df2ea62018dba1ffef834993d1e38549648d6941"}
Nov 03 23:52:35.533564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.533536    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:35.575129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.575096    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx\" (UniqueName: \"kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx\") pod \"8baf683e-b684-479c-9a75-7e405002c4a2\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") "
Nov 03 23:52:35.575310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.575225    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca\") pod \"8baf683e-b684-479c-9a75-7e405002c4a2\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") "
Nov 03 23:52:35.575310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.575307    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca\") pod \"8baf683e-b684-479c-9a75-7e405002c4a2\" (UID: \"8baf683e-b684-479c-9a75-7e405002c4a2\") "
Nov 03 23:52:35.575798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.575764    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca" pod "8baf683e-b684-479c-9a75-7e405002c4a2" (UID: "8baf683e-b684-479c-9a75-7e405002c4a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:35.577803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.577777    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx" pod "8baf683e-b684-479c-9a75-7e405002c4a2" (UID: "8baf683e-b684-479c-9a75-7e405002c4a2"). InnerVolumeSpecName "kube-api-access-glklx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:35.580425 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.580395    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca" pod "8baf683e-b684-479c-9a75-7e405002c4a2" (UID: "8baf683e-b684-479c-9a75-7e405002c4a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:35.677572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.677484    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:35.677572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.677515    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baf683e-b684-479c-9a75-7e405002c4a2-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:35.677572 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:35.677527    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glklx\" (UniqueName: \"kubernetes.io/projected/8baf683e-b684-479c-9a75-7e405002c4a2-kube-api-access-glklx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.076563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.076499    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"]
Nov 03 23:52:36.077389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.077345    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2" podUID="f209157f-4001-47b8-bf92-293790d63df4" containerName="payload-processing" containerID="cri-o://21a95a93bdd4f59f8c8acdcf66a75cb2d651f9c4220dcab6486b41766ff6cfb9" gracePeriod=30
Nov 03 23:52:36.086048 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.086016    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"]
Nov 03 23:52:36.086452 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.086380    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp" podUID="d27468ed-ec84-419d-8e12-a5bf8e3e4e80" containerName="payload-pre-processing" containerID="cri-o://45245ecfa62fc0d854c66063afbdc0ae5bc5e02582771d5c2c0e910bff87b726" gracePeriod=30
Nov 03 23:52:36.455793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.445058    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v" event={"ID":"8baf683e-b684-479c-9a75-7e405002c4a2","Type":"ContainerDied","Data":"f4ae92e36446e65b22031651c335571bec5bcaa6296cb5dbf7781fe2a71ca2e0"}
Nov 03 23:52:36.455793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.445127    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ae92e36446e65b22031651c335571bec5bcaa6296cb5dbf7781fe2a71ca2e0"
Nov 03 23:52:36.455793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.445206    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"
Nov 03 23:52:36.455793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.455432    5074 generic.go:356] "Generic (PLEG): container finished" podID="f209157f-4001-47b8-bf92-293790d63df4" containerID="21a95a93bdd4f59f8c8acdcf66a75cb2d651f9c4220dcab6486b41766ff6cfb9" exitCode=0
Nov 03 23:52:36.460807 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.455725    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2" event={"ID":"f209157f-4001-47b8-bf92-293790d63df4","Type":"ContainerDied","Data":"21a95a93bdd4f59f8c8acdcf66a75cb2d651f9c4220dcab6486b41766ff6cfb9"}
Nov 03 23:52:36.462451 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.462422    5074 generic.go:356] "Generic (PLEG): container finished" podID="d27468ed-ec84-419d-8e12-a5bf8e3e4e80" containerID="45245ecfa62fc0d854c66063afbdc0ae5bc5e02582771d5c2c0e910bff87b726" exitCode=0
Nov 03 23:52:36.462531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.462463    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp" event={"ID":"d27468ed-ec84-419d-8e12-a5bf8e3e4e80","Type":"ContainerDied","Data":"45245ecfa62fc0d854c66063afbdc0ae5bc5e02582771d5c2c0e910bff87b726"}
Nov 03 23:52:36.541619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.541590    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:52:36.633286 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.633258    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:52:36.636272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.636238    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume\") pod \"f209157f-4001-47b8-bf92-293790d63df4\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") "
Nov 03 23:52:36.636574 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.636552    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv\" (UniqueName: \"kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv\") pod \"f209157f-4001-47b8-bf92-293790d63df4\" (UID: \"f209157f-4001-47b8-bf92-293790d63df4\") "
Nov 03 23:52:36.637445 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.637419    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume" pod "f209157f-4001-47b8-bf92-293790d63df4" (UID: "f209157f-4001-47b8-bf92-293790d63df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:36.640911 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.640876    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv" pod "f209157f-4001-47b8-bf92-293790d63df4" (UID: "f209157f-4001-47b8-bf92-293790d63df4"). InnerVolumeSpecName "kube-api-access-mzqrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:36.738480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.738388    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp\" (UniqueName: \"kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp\") pod \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") "
Nov 03 23:52:36.738480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.738435    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume\") pod \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\" (UID: \"d27468ed-ec84-419d-8e12-a5bf8e3e4e80\") "
Nov 03 23:52:36.738917 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.738899    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzqrv\" (UniqueName: \"kubernetes.io/projected/f209157f-4001-47b8-bf92-293790d63df4-kube-api-access-mzqrv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.739011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.738920    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f209157f-4001-47b8-bf92-293790d63df4-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.739011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.738918    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume" pod "d27468ed-ec84-419d-8e12-a5bf8e3e4e80" (UID: "d27468ed-ec84-419d-8e12-a5bf8e3e4e80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:36.740611 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.740581    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp" pod "d27468ed-ec84-419d-8e12-a5bf8e3e4e80" (UID: "d27468ed-ec84-419d-8e12-a5bf8e3e4e80"). InnerVolumeSpecName "kube-api-access-wkmhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:36.840293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.840255    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkmhp\" (UniqueName: \"kubernetes.io/projected/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-kube-api-access-wkmhp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.840293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:36.840295    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27468ed-ec84-419d-8e12-a5bf8e3e4e80-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:37.030778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.030715    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"]
Nov 03 23:52:37.034332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.034304    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824ba939-zsvp2"]
Nov 03 23:52:37.471205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.471174    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"
Nov 03 23:52:37.471381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.471165    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp" event={"ID":"d27468ed-ec84-419d-8e12-a5bf8e3e4e80","Type":"ContainerDied","Data":"fbc55dfbc00f72dcf8da0b181b37da265e5c352d5312b8d211cf576f714cd786"}
Nov 03 23:52:37.471381 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.471317    5074 scope.go:122] "RemoveContainer" containerID="45245ecfa62fc0d854c66063afbdc0ae5bc5e02582771d5c2c0e910bff87b726"
Nov 03 23:52:37.472824 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.472801    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"
Nov 03 23:52:37.472952 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.472843    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2" event={"ID":"f209157f-4001-47b8-bf92-293790d63df4","Type":"ContainerDied","Data":"b29bf1b53679c0430dbc1f9b272c3734003b90d3595291ca21436a7a6ea83330"}
Nov 03 23:52:37.482195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.482175    5074 scope.go:122] "RemoveContainer" containerID="21a95a93bdd4f59f8c8acdcf66a75cb2d651f9c4220dcab6486b41766ff6cfb9"
Nov 03 23:52:37.496384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.496356    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"]
Nov 03 23:52:37.501033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.501012    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-f7c55d7bd-cjvx2"]
Nov 03 23:52:37.513703 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.513679    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"]
Nov 03 23:52:37.517802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:37.517777    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-67b478c77f-k8cdp"]
Nov 03 23:52:39.018986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:39.018947    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2bada4-b700-4183-af93-093d8a6ed0b4" path="/var/lib/kubelet/pods/3b2bada4-b700-4183-af93-093d8a6ed0b4/volumes"
Nov 03 23:52:39.019579 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:39.019557    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27468ed-ec84-419d-8e12-a5bf8e3e4e80" path="/var/lib/kubelet/pods/d27468ed-ec84-419d-8e12-a5bf8e3e4e80/volumes"
Nov 03 23:52:39.020085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:39.020064    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f209157f-4001-47b8-bf92-293790d63df4" path="/var/lib/kubelet/pods/f209157f-4001-47b8-bf92-293790d63df4/volumes"
Nov 03 23:52:40.030978 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:40.030945    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"]
Nov 03 23:52:40.034800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:40.034768    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f338e4-5mmq7"]
Nov 03 23:52:41.019238 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:41.019206    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d7eee2-545e-475f-9ea5-8a705b1aae06" path="/var/lib/kubelet/pods/98d7eee2-545e-475f-9ea5-8a705b1aae06/volumes"
Nov 03 23:52:43.559528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:43.559484    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"]
Nov 03 23:52:43.560544 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:43.560484    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2" podUID="c59ba856-3f73-4ff7-93e3-ef3b64b37548" containerName="payload-pre-processing" containerID="cri-o://6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88" gracePeriod=30
Nov 03 23:52:44.061554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.061520    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:52:44.167080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.166986    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q\" (UniqueName: \"kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q\") pod \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") "
Nov 03 23:52:44.167259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.167201    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume\") pod \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\" (UID: \"c59ba856-3f73-4ff7-93e3-ef3b64b37548\") "
Nov 03 23:52:44.167691 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.167660    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume" pod "c59ba856-3f73-4ff7-93e3-ef3b64b37548" (UID: "c59ba856-3f73-4ff7-93e3-ef3b64b37548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:44.167839 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.167815    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59ba856-3f73-4ff7-93e3-ef3b64b37548-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.169260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.169233    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q" pod "c59ba856-3f73-4ff7-93e3-ef3b64b37548" (UID: "c59ba856-3f73-4ff7-93e3-ef3b64b37548"). InnerVolumeSpecName "kube-api-access-nll8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:44.269766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.269692    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nll8q\" (UniqueName: \"kubernetes.io/projected/c59ba856-3f73-4ff7-93e3-ef3b64b37548-kube-api-access-nll8q\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.536318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.536275    5074 generic.go:356] "Generic (PLEG): container finished" podID="c59ba856-3f73-4ff7-93e3-ef3b64b37548" containerID="6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88" exitCode=0
Nov 03 23:52:44.536523 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.536327    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"
Nov 03 23:52:44.536523 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.536362    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2" event={"ID":"c59ba856-3f73-4ff7-93e3-ef3b64b37548","Type":"ContainerDied","Data":"6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88"}
Nov 03 23:52:44.536523 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.536446    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2" event={"ID":"c59ba856-3f73-4ff7-93e3-ef3b64b37548","Type":"ContainerDied","Data":"a91d10038f020cc5616588d13c6e403b5fc491697a9cb635b770053979c84771"}
Nov 03 23:52:44.536523 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.536485    5074 scope.go:122] "RemoveContainer" containerID="6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88"
Nov 03 23:52:44.549299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.549280    5074 scope.go:122] "RemoveContainer" containerID="6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88"
Nov 03 23:52:44.549628 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:52:44.549599    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88\": container with ID starting with 6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88 not found: ID does not exist" containerID="6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88"
Nov 03 23:52:44.549766 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.549633    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88"} err="failed to get container status \"6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88\": rpc error: code = NotFound desc = could not find container \"6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88\": container with ID starting with 6e61cf8bdcdc0d53700fd3c868fd666d7d0f39356956835af6edef4be8fe3c88 not found: ID does not exist"
Nov 03 23:52:44.564472 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.564446    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"]
Nov 03 23:52:44.570794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.570770    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-5b6858c4dc-rzfk2"]
Nov 03 23:52:44.780885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.780854    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"]
Nov 03 23:52:44.781480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781465    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8baf683e-b684-479c-9a75-7e405002c4a2" containerName="revoke-keys"
Nov 03 23:52:44.781528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781482    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8baf683e-b684-479c-9a75-7e405002c4a2" containerName="revoke-keys"
Nov 03 23:52:44.781528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781496    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d27468ed-ec84-419d-8e12-a5bf8e3e4e80" containerName="payload-pre-processing"
Nov 03 23:52:44.781528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781501    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27468ed-ec84-419d-8e12-a5bf8e3e4e80" containerName="payload-pre-processing"
Nov 03 23:52:44.781528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781517    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c59ba856-3f73-4ff7-93e3-ef3b64b37548" containerName="payload-pre-processing"
Nov 03 23:52:44.781528 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781526    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59ba856-3f73-4ff7-93e3-ef3b64b37548" containerName="payload-pre-processing"
Nov 03 23:52:44.781671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781554    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f209157f-4001-47b8-bf92-293790d63df4" containerName="payload-processing"
Nov 03 23:52:44.781671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781559    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f209157f-4001-47b8-bf92-293790d63df4" containerName="payload-processing"
Nov 03 23:52:44.781671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781644    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f209157f-4001-47b8-bf92-293790d63df4" containerName="payload-processing"
Nov 03 23:52:44.781671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781657    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8baf683e-b684-479c-9a75-7e405002c4a2" containerName="revoke-keys"
Nov 03 23:52:44.781671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781664    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27468ed-ec84-419d-8e12-a5bf8e3e4e80" containerName="payload-pre-processing"
Nov 03 23:52:44.781671 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.781670    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59ba856-3f73-4ff7-93e3-ef3b64b37548" containerName="payload-pre-processing"
Nov 03 23:52:44.786589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.786530    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.792635 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.792614    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"]
Nov 03 23:52:44.793005 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.792985    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:52:44.795176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.795143    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:52:44.875728 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.875698    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.875960 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.875733    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.876077 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.876056    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj55f\" (UniqueName: \"kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.976804 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.976767    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj55f\" (UniqueName: \"kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.976958 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.976854    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.976958 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.976871    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.977507 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.977480    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.979321 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.979301    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:44.989652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:44.989634    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj55f\" (UniqueName: \"kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f\") pod \"maas-api-revoke-sub-8a89227a-rxvkg\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:45.026874 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:45.026841    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59ba856-3f73-4ff7-93e3-ef3b64b37548" path="/var/lib/kubelet/pods/c59ba856-3f73-4ff7-93e3-ef3b64b37548/volumes"
Nov 03 23:52:45.116812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:45.116428    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:45.416058 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:45.412458    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"]
Nov 03 23:52:45.547353 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:45.547315    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg" event={"ID":"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c","Type":"ContainerStarted","Data":"42479e706fe331996b0c2a9a39702b56903d1d055cc64eb5032002d8b61287b6"}
Nov 03 23:52:46.252939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.252900    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"]
Nov 03 23:52:46.257503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.257479    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.277881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.271644    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"]
Nov 03 23:52:46.297152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.295556    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.297152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.295652    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.297152 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.295707    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpgn\" (UniqueName: \"kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.397604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.397565    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.397823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.397645    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.397823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.397695    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpgn\" (UniqueName: \"kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.398372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.398337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.400231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.400178    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.410822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.410797    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpgn\" (UniqueName: \"kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn\") pod \"maas-api-revoke-sub-478eb918-t87r7\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.556920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.556832    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" containerID="e330643b5ca53e30a67219f9114b7844974a05ae0e8628e160482e8a4152ec20" exitCode=0
Nov 03 23:52:46.557080 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.556914    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg" event={"ID":"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c","Type":"ContainerDied","Data":"e330643b5ca53e30a67219f9114b7844974a05ae0e8628e160482e8a4152ec20"}
Nov 03 23:52:46.636078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.636043    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:46.923268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:46.923182    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"]
Nov 03 23:52:46.930346 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:52:46.930312    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6e0576_b847_49c6_b9e6_0de41645887a.slice/crio-a427e811e961d82339c24835d96208438ac3b56ed38a3b52348aad79521f119d WatchSource:0}: Error finding container a427e811e961d82339c24835d96208438ac3b56ed38a3b52348aad79521f119d: Status 404 returned error can't find the container with id a427e811e961d82339c24835d96208438ac3b56ed38a3b52348aad79521f119d
Nov 03 23:52:47.566252 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.566148    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7" event={"ID":"fb6e0576-b847-49c6-b9e6-0de41645887a","Type":"ContainerStarted","Data":"d1fafe77560d380c0f7ec9eec82bd82032145b9c3bc16fc4f9f8069163865d3d"}
Nov 03 23:52:47.566252 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.566216    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7" event={"ID":"fb6e0576-b847-49c6-b9e6-0de41645887a","Type":"ContainerStarted","Data":"a427e811e961d82339c24835d96208438ac3b56ed38a3b52348aad79521f119d"}
Nov 03 23:52:47.587718 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.587658    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7" podStartSLOduration=1.587640064 podStartE2EDuration="1.587640064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:47.584972746 +0000 UTC m=+1432.684572893" watchObservedRunningTime="2026-11-03 23:52:47.587640064 +0000 UTC m=+1432.687240212"
Nov 03 23:52:47.744191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.744167    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:47.828421 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.828348    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca\") pod \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") "
Nov 03 23:52:47.828578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.828435    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca\") pod \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") "
Nov 03 23:52:47.828578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.828530    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f\" (UniqueName: \"kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f\") pod \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\" (UID: \"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c\") "
Nov 03 23:52:47.828791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.828769    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca" pod "4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" (UID: "4d0a0f5d-ef70-45cf-bca9-92f09faeee6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:47.829115 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.829087    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:47.830827 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.830803    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca" pod "4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" (UID: "4d0a0f5d-ef70-45cf-bca9-92f09faeee6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:47.830945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.830915    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f" pod "4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" (UID: "4d0a0f5d-ef70-45cf-bca9-92f09faeee6c"). InnerVolumeSpecName "kube-api-access-fj55f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:47.930320 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.930295    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:47.930320 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:47.930322    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj55f\" (UniqueName: \"kubernetes.io/projected/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c-kube-api-access-fj55f\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:48.013444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.013411    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"]
Nov 03 23:52:48.014177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.014157    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" containerName="revoke-keys"
Nov 03 23:52:48.014177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.014178    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" containerName="revoke-keys"
Nov 03 23:52:48.014306 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.014283    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" containerName="revoke-keys"
Nov 03 23:52:48.017708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.017689    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.026031 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.026000    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"]
Nov 03 23:52:48.144197 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.144108    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.144197 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.144174    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75h6\" (UniqueName: \"kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.144904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.144882    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.246466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.246422    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.246646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.246490    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.246646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.246523    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75h6\" (UniqueName: \"kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.247231 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.247208    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.248996 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.248964    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.260555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.260527    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75h6\" (UniqueName: \"kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6\") pod \"maas-api-revoke-sub-73ae5419-cp7fx\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.350811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.350780    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:48.575325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.575297    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"
Nov 03 23:52:48.575835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.575336    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg" event={"ID":"4d0a0f5d-ef70-45cf-bca9-92f09faeee6c","Type":"ContainerDied","Data":"42479e706fe331996b0c2a9a39702b56903d1d055cc64eb5032002d8b61287b6"}
Nov 03 23:52:48.575835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.575382    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42479e706fe331996b0c2a9a39702b56903d1d055cc64eb5032002d8b61287b6"
Nov 03 23:52:48.577273 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.577248    5074 generic.go:356] "Generic (PLEG): container finished" podID="fb6e0576-b847-49c6-b9e6-0de41645887a" containerID="d1fafe77560d380c0f7ec9eec82bd82032145b9c3bc16fc4f9f8069163865d3d" exitCode=0
Nov 03 23:52:48.577376 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.577333    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7" event={"ID":"fb6e0576-b847-49c6-b9e6-0de41645887a","Type":"ContainerDied","Data":"d1fafe77560d380c0f7ec9eec82bd82032145b9c3bc16fc4f9f8069163865d3d"}
Nov 03 23:52:48.637953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:48.637904    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"]
Nov 03 23:52:48.638218 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:52:48.638177    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389c764a_caab_4391_96b7_91fa6d2362cd.slice/crio-2f958f7cd6b649cd95ad2e4d4352e573558d5b8d6afb9459184591427bbef36b WatchSource:0}: Error finding container 2f958f7cd6b649cd95ad2e4d4352e573558d5b8d6afb9459184591427bbef36b: Status 404 returned error can't find the container with id 2f958f7cd6b649cd95ad2e4d4352e573558d5b8d6afb9459184591427bbef36b
Nov 03 23:52:49.586079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.586046    5074 generic.go:356] "Generic (PLEG): container finished" podID="389c764a-caab-4391-96b7-91fa6d2362cd" containerID="5087397d56ca7a4eeba18f26113c9b74f77384c0ff2ffd85fc7d9b0bed3c891a" exitCode=0
Nov 03 23:52:49.586459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.586123    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx" event={"ID":"389c764a-caab-4391-96b7-91fa6d2362cd","Type":"ContainerDied","Data":"5087397d56ca7a4eeba18f26113c9b74f77384c0ff2ffd85fc7d9b0bed3c891a"}
Nov 03 23:52:49.586459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.586178    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx" event={"ID":"389c764a-caab-4391-96b7-91fa6d2362cd","Type":"ContainerStarted","Data":"2f958f7cd6b649cd95ad2e4d4352e573558d5b8d6afb9459184591427bbef36b"}
Nov 03 23:52:49.785721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.785693    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:49.868678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.868636    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca\") pod \"fb6e0576-b847-49c6-b9e6-0de41645887a\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") "
Nov 03 23:52:49.868892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.868696    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca\") pod \"fb6e0576-b847-49c6-b9e6-0de41645887a\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") "
Nov 03 23:52:49.868892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.868852    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn\" (UniqueName: \"kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn\") pod \"fb6e0576-b847-49c6-b9e6-0de41645887a\" (UID: \"fb6e0576-b847-49c6-b9e6-0de41645887a\") "
Nov 03 23:52:49.870406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.870373    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca" pod "fb6e0576-b847-49c6-b9e6-0de41645887a" (UID: "fb6e0576-b847-49c6-b9e6-0de41645887a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:49.873861 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.872895    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn" pod "fb6e0576-b847-49c6-b9e6-0de41645887a" (UID: "fb6e0576-b847-49c6-b9e6-0de41645887a"). InnerVolumeSpecName "kube-api-access-ndpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:49.894375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.889936    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca" pod "fb6e0576-b847-49c6-b9e6-0de41645887a" (UID: "fb6e0576-b847-49c6-b9e6-0de41645887a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:49.972760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.972141    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:49.972760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.972180    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e0576-b847-49c6-b9e6-0de41645887a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:49.972760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:49.972197    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndpgn\" (UniqueName: \"kubernetes.io/projected/fb6e0576-b847-49c6-b9e6-0de41645887a-kube-api-access-ndpgn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:50.379779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.377648    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"]
Nov 03 23:52:50.379779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.378728    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fb6e0576-b847-49c6-b9e6-0de41645887a" containerName="revoke-keys"
Nov 03 23:52:50.379779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.379273    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6e0576-b847-49c6-b9e6-0de41645887a" containerName="revoke-keys"
Nov 03 23:52:50.379779 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.379515    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6e0576-b847-49c6-b9e6-0de41645887a" containerName="revoke-keys"
Nov 03 23:52:50.396066 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.396036    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.420096 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.420067    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"]
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489450    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489536    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489589    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489638    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2kb\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489684    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489768    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.489791 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489794    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.490264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489832    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.490264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.489955    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.591787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.591732    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.591802    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.591847    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.591956    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.592067    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.592124    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.592181    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.592232    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2kb\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592532 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.592284    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.592815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.592789    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.594029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.593999    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.594171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.594124    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.596647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.595606    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.596647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.596115    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.605770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.600913    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"
Nov 03 23:52:50.605770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.602972    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.612769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.606396    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.612769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.610240    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.612769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.610305    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7" event={"ID":"fb6e0576-b847-49c6-b9e6-0de41645887a","Type":"ContainerDied","Data":"a427e811e961d82339c24835d96208438ac3b56ed38a3b52348aad79521f119d"}
Nov 03 23:52:50.612769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.610355    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a427e811e961d82339c24835d96208438ac3b56ed38a3b52348aad79521f119d"
Nov 03 23:52:50.646875 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.619373    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2kb\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.747908 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.747649    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:50.814038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.814012    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:50.899446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.898059    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca\") pod \"389c764a-caab-4391-96b7-91fa6d2362cd\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") "
Nov 03 23:52:50.899446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.898207    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca\") pod \"389c764a-caab-4391-96b7-91fa6d2362cd\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") "
Nov 03 23:52:50.899446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.898277    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6\" (UniqueName: \"kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6\") pod \"389c764a-caab-4391-96b7-91fa6d2362cd\" (UID: \"389c764a-caab-4391-96b7-91fa6d2362cd\") "
Nov 03 23:52:50.899446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.899344    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca" pod "389c764a-caab-4391-96b7-91fa6d2362cd" (UID: "389c764a-caab-4391-96b7-91fa6d2362cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:50.902967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.902924    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6" pod "389c764a-caab-4391-96b7-91fa6d2362cd" (UID: "389c764a-caab-4391-96b7-91fa6d2362cd"). InnerVolumeSpecName "kube-api-access-g75h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:50.903195 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:50.903165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca" pod "389c764a-caab-4391-96b7-91fa6d2362cd" (UID: "389c764a-caab-4391-96b7-91fa6d2362cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:51.014772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.004553    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:51.014772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.004594    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389c764a-caab-4391-96b7-91fa6d2362cd-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:51.014772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.004611    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75h6\" (UniqueName: \"kubernetes.io/projected/389c764a-caab-4391-96b7-91fa6d2362cd-kube-api-access-g75h6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:51.080531 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.080502    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"]
Nov 03 23:52:51.089469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.085240    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:52:51.089469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.085314    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:52:51.089469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.085354    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:52:51.675432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.674403    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" event={"ID":"e46008f3-d315-42f4-b9fc-79acf87fb9cd","Type":"ContainerStarted","Data":"47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8"}
Nov 03 23:52:51.675432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.674466    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" event={"ID":"e46008f3-d315-42f4-b9fc-79acf87fb9cd","Type":"ContainerStarted","Data":"98b797c9d1d2b0ab6411c5a2eb5e40ec1285c9981bc1b8377ff9b1763f48062c"}
Nov 03 23:52:51.676649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.676619    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx" event={"ID":"389c764a-caab-4391-96b7-91fa6d2362cd","Type":"ContainerDied","Data":"2f958f7cd6b649cd95ad2e4d4352e573558d5b8d6afb9459184591427bbef36b"}
Nov 03 23:52:51.676895 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.676866    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f958f7cd6b649cd95ad2e4d4352e573558d5b8d6afb9459184591427bbef36b"
Nov 03 23:52:51.677009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.676946    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"
Nov 03 23:52:51.755763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.754848    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:51.768819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:51.755970    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podStartSLOduration=1.755952207 podStartE2EDuration="1.755952207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:51.706166487 +0000 UTC m=+1436.805766632" watchObservedRunningTime="2026-11-03 23:52:51.755952207 +0000 UTC m=+1436.855552351"
Nov 03 23:52:52.751380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:52.751347    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:52.751787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:52.751410    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:53.751771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:53.751211    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:53.751771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:53.751315    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:54.749194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:54.749154    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:54.749408 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:54.749225    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:55.749198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:55.749146    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:55.749695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:55.749207    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:56.151144 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.151061    5074 scope.go:122] "RemoveContainer" containerID="e967ac97cd7ddb6123321f799900623231ab1cab17e9f5930b589489a7088d2d"
Nov 03 23:52:56.582453 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.582419    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"]
Nov 03 23:52:56.583109 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.583092    5074 cpu_manager.go:414] "containerMap: removing container" podUID="389c764a-caab-4391-96b7-91fa6d2362cd" containerName="revoke-keys"
Nov 03 23:52:56.583176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.583110    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389c764a-caab-4391-96b7-91fa6d2362cd" containerName="revoke-keys"
Nov 03 23:52:56.583279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.583265    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="389c764a-caab-4391-96b7-91fa6d2362cd" containerName="revoke-keys"
Nov 03 23:52:56.590563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.590538    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.595379 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.595352    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:52:56.596978 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.596958    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:52:56.597272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.597253    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"]
Nov 03 23:52:56.755078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.754553    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:56.755078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.754633    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:56.761324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.761278    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44m6\" (UniqueName: \"kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.761492 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.761413    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.761492 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.761451    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.862857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.862765    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44m6\" (UniqueName: \"kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.863021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.862908    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.863021 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.862951    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.863972 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.863945    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.868641 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.868610    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.875689 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.875667    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44m6\" (UniqueName: \"kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6\") pod \"maas-api-revoke-sub-68b66921-zlcws\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:56.962804 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:56.962772    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:52:57.360006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:57.359979    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"]
Nov 03 23:52:57.362530 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:52:57.361897    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe453ec_b910_4233_a3ed_43f3507cf73a.slice/crio-18e1441bedcd7e191ca85eb7177bcde9316a574f53f420b8efa96c09cd12237f WatchSource:0}: Error finding container 18e1441bedcd7e191ca85eb7177bcde9316a574f53f420b8efa96c09cd12237f: Status 404 returned error can't find the container with id 18e1441bedcd7e191ca85eb7177bcde9316a574f53f420b8efa96c09cd12237f
Nov 03 23:52:57.749153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:57.749120    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:57.749266 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:57.749183    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:57.752816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:57.752765    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws" event={"ID":"7fe453ec-b910-4233-a3ed-43f3507cf73a","Type":"ContainerStarted","Data":"18e1441bedcd7e191ca85eb7177bcde9316a574f53f420b8efa96c09cd12237f"}
Nov 03 23:52:58.047204 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:52:58.047116    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:52:58.749328 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:58.749293    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 23:52:58.749567 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:58.749368    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 23:52:58.763707 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:58.763677    5074 generic.go:356] "Generic (PLEG): container finished" podID="7fe453ec-b910-4233-a3ed-43f3507cf73a" containerID="95d92d1f4503497bdfa35aa1c79ddbe6cf3c309e2f25d9be0348d673b6f5918a" exitCode=0
Nov 03 23:52:58.763872 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:58.763775    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws" event={"ID":"7fe453ec-b910-4233-a3ed-43f3507cf73a","Type":"ContainerDied","Data":"95d92d1f4503497bdfa35aa1c79ddbe6cf3c309e2f25d9be0348d673b6f5918a"}
Nov 03 23:52:59.753855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:59.753821    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:59.754348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:59.754276    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:52:59.755131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:52:59.755106    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 03 23:53:00.083948 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.083917    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:53:00.122654 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.122623    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6\" (UniqueName: \"kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6\") pod \"7fe453ec-b910-4233-a3ed-43f3507cf73a\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") "
Nov 03 23:53:00.127131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.127093    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6" pod "7fe453ec-b910-4233-a3ed-43f3507cf73a" (UID: "7fe453ec-b910-4233-a3ed-43f3507cf73a"). InnerVolumeSpecName "kube-api-access-p44m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:00.224177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.224116    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca\") pod \"7fe453ec-b910-4233-a3ed-43f3507cf73a\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") "
Nov 03 23:53:00.224389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.224240    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca\") pod \"7fe453ec-b910-4233-a3ed-43f3507cf73a\" (UID: \"7fe453ec-b910-4233-a3ed-43f3507cf73a\") "
Nov 03 23:53:00.227114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.227047    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca" pod "7fe453ec-b910-4233-a3ed-43f3507cf73a" (UID: "7fe453ec-b910-4233-a3ed-43f3507cf73a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:00.227313 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.227175    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44m6\" (UniqueName: \"kubernetes.io/projected/7fe453ec-b910-4233-a3ed-43f3507cf73a-kube-api-access-p44m6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.229191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.229159    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca" pod "7fe453ec-b910-4233-a3ed-43f3507cf73a" (UID: "7fe453ec-b910-4233-a3ed-43f3507cf73a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:00.328344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.328262    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.328344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.328296    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe453ec-b910-4233-a3ed-43f3507cf73a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.539723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.539689    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"]
Nov 03 23:53:00.540370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.540351    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7fe453ec-b910-4233-a3ed-43f3507cf73a" containerName="revoke-keys"
Nov 03 23:53:00.540370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.540369    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe453ec-b910-4233-a3ed-43f3507cf73a" containerName="revoke-keys"
Nov 03 23:53:00.540524 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.540479    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe453ec-b910-4233-a3ed-43f3507cf73a" containerName="revoke-keys"
Nov 03 23:53:00.544324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.544303    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.555239 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.555212    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"]
Nov 03 23:53:00.737672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.737632    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.737885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.737826    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.737885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.737866    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twclc\" (UniqueName: \"kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.781302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.781257    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"]
Nov 03 23:53:00.787651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.787610    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.793814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.793783    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"]
Nov 03 23:53:00.795799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.795769    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"
Nov 03 23:53:00.795906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.795772    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws" event={"ID":"7fe453ec-b910-4233-a3ed-43f3507cf73a","Type":"ContainerDied","Data":"18e1441bedcd7e191ca85eb7177bcde9316a574f53f420b8efa96c09cd12237f"}
Nov 03 23:53:00.795906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.795849    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e1441bedcd7e191ca85eb7177bcde9316a574f53f420b8efa96c09cd12237f"
Nov 03 23:53:00.838931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.838897    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.839120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.839030    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.839120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.839059    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8brx\" (UniqueName: \"kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.839120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.839077    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.839120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.839097    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.839370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.839255    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twclc\" (UniqueName: \"kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.839660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.839634    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.841438 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.841415    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.852086 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.852065    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twclc\" (UniqueName: \"kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc\") pod \"maas-api-revoke-sub-48b41dec-tzkrb\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.878008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.877975    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:00.940793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.940753    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.940989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.940807    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8brx\" (UniqueName: \"kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.940989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.940830    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.941612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.941586    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.943256 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.943230    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:00.957342 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:00.957317    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8brx\" (UniqueName: \"kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx\") pod \"maas-api-revoke-sub-8eb060ae-bqkbg\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:01.108388 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.108277    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:01.191673 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:01.191634    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813afa38_726c_48a5_9b99_c47ac49db7ad.slice/crio-1d56256e524dbf2dce2708d33887cf885b1725b8a0bebc82f9d27e0f88220937 WatchSource:0}: Error finding container 1d56256e524dbf2dce2708d33887cf885b1725b8a0bebc82f9d27e0f88220937: Status 404 returned error can't find the container with id 1d56256e524dbf2dce2708d33887cf885b1725b8a0bebc82f9d27e0f88220937
Nov 03 23:53:01.192075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.192043    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"]
Nov 03 23:53:01.396036 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:01.396008    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa51b005_48b9_49d9_91a2_182dff56564a.slice/crio-7268566181f611a97e11f6e91469fc5c1da37f41f7191ffd1e9d63342112b72d WatchSource:0}: Error finding container 7268566181f611a97e11f6e91469fc5c1da37f41f7191ffd1e9d63342112b72d: Status 404 returned error can't find the container with id 7268566181f611a97e11f6e91469fc5c1da37f41f7191ffd1e9d63342112b72d
Nov 03 23:53:01.398135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.398107    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"]
Nov 03 23:53:01.806219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.806178    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg" event={"ID":"fa51b005-48b9-49d9-91a2-182dff56564a","Type":"ContainerStarted","Data":"439af23996438adbdf26cb40709a5365ff34ce2d0b263b8308b1f029eac6a90f"}
Nov 03 23:53:01.806691 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.806247    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg" event={"ID":"fa51b005-48b9-49d9-91a2-182dff56564a","Type":"ContainerStarted","Data":"7268566181f611a97e11f6e91469fc5c1da37f41f7191ffd1e9d63342112b72d"}
Nov 03 23:53:01.808193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.808163    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb" event={"ID":"813afa38-726c-48a5-9b99-c47ac49db7ad","Type":"ContainerStarted","Data":"1f597c41fc6d5121909344fadff8b81cf30638b35153bb0f090a1b15223992c8"}
Nov 03 23:53:01.808307 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.808219    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb" event={"ID":"813afa38-726c-48a5-9b99-c47ac49db7ad","Type":"ContainerStarted","Data":"1d56256e524dbf2dce2708d33887cf885b1725b8a0bebc82f9d27e0f88220937"}
Nov 03 23:53:01.829598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.829554    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg" podStartSLOduration=1.8295406459999999 podStartE2EDuration="1.829540646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:01.826106985 +0000 UTC m=+1446.925707130" watchObservedRunningTime="2026-11-03 23:53:01.829540646 +0000 UTC m=+1446.929140790"
Nov 03 23:53:01.848263 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:01.848221    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb" podStartSLOduration=1.84820793 podStartE2EDuration="1.84820793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:01.847006063 +0000 UTC m=+1446.946606208" watchObservedRunningTime="2026-11-03 23:53:01.84820793 +0000 UTC m=+1446.947808103"
Nov 03 23:53:02.817477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:02.817439    5074 generic.go:356] "Generic (PLEG): container finished" podID="fa51b005-48b9-49d9-91a2-182dff56564a" containerID="439af23996438adbdf26cb40709a5365ff34ce2d0b263b8308b1f029eac6a90f" exitCode=0
Nov 03 23:53:02.817945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:02.817526    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg" event={"ID":"fa51b005-48b9-49d9-91a2-182dff56564a","Type":"ContainerDied","Data":"439af23996438adbdf26cb40709a5365ff34ce2d0b263b8308b1f029eac6a90f"}
Nov 03 23:53:02.819218 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:02.819193    5074 generic.go:356] "Generic (PLEG): container finished" podID="813afa38-726c-48a5-9b99-c47ac49db7ad" containerID="1f597c41fc6d5121909344fadff8b81cf30638b35153bb0f090a1b15223992c8" exitCode=0
Nov 03 23:53:02.819332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:02.819276    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb" event={"ID":"813afa38-726c-48a5-9b99-c47ac49db7ad","Type":"ContainerDied","Data":"1f597c41fc6d5121909344fadff8b81cf30638b35153bb0f090a1b15223992c8"}
Nov 03 23:53:04.048821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.048683    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:04.084205 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.084133    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:04.116561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.116529    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca\") pod \"fa51b005-48b9-49d9-91a2-182dff56564a\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") "
Nov 03 23:53:04.116783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.116619    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca\") pod \"fa51b005-48b9-49d9-91a2-182dff56564a\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") "
Nov 03 23:53:04.116783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.116662    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx\" (UniqueName: \"kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx\") pod \"fa51b005-48b9-49d9-91a2-182dff56564a\" (UID: \"fa51b005-48b9-49d9-91a2-182dff56564a\") "
Nov 03 23:53:04.116783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.116693    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca\") pod \"813afa38-726c-48a5-9b99-c47ac49db7ad\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") "
Nov 03 23:53:04.117190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.117161    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca" pod "813afa38-726c-48a5-9b99-c47ac49db7ad" (UID: "813afa38-726c-48a5-9b99-c47ac49db7ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:04.117291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.116812    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca\") pod \"813afa38-726c-48a5-9b99-c47ac49db7ad\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") "
Nov 03 23:53:04.117335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.117310    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc\" (UniqueName: \"kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc\") pod \"813afa38-726c-48a5-9b99-c47ac49db7ad\" (UID: \"813afa38-726c-48a5-9b99-c47ac49db7ad\") "
Nov 03 23:53:04.117380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.117359    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca" pod "fa51b005-48b9-49d9-91a2-182dff56564a" (UID: "fa51b005-48b9-49d9-91a2-182dff56564a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:04.118503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.118138    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:04.118503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.118163    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:04.121288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.119798    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc" pod "813afa38-726c-48a5-9b99-c47ac49db7ad" (UID: "813afa38-726c-48a5-9b99-c47ac49db7ad"). InnerVolumeSpecName "kube-api-access-twclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:04.121288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.120711    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca" pod "813afa38-726c-48a5-9b99-c47ac49db7ad" (UID: "813afa38-726c-48a5-9b99-c47ac49db7ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:04.121712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.121675    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx" pod "fa51b005-48b9-49d9-91a2-182dff56564a" (UID: "fa51b005-48b9-49d9-91a2-182dff56564a"). InnerVolumeSpecName "kube-api-access-n8brx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:04.122156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.122131    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca" pod "fa51b005-48b9-49d9-91a2-182dff56564a" (UID: "fa51b005-48b9-49d9-91a2-182dff56564a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:04.219170 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.219127    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8brx\" (UniqueName: \"kubernetes.io/projected/fa51b005-48b9-49d9-91a2-182dff56564a-kube-api-access-n8brx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:04.219170 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.219161    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813afa38-726c-48a5-9b99-c47ac49db7ad-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:04.219170 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.219176    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twclc\" (UniqueName: \"kubernetes.io/projected/813afa38-726c-48a5-9b99-c47ac49db7ad-kube-api-access-twclc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:04.219399 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.219190    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa51b005-48b9-49d9-91a2-182dff56564a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:04.848763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.841954    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg" event={"ID":"fa51b005-48b9-49d9-91a2-182dff56564a","Type":"ContainerDied","Data":"7268566181f611a97e11f6e91469fc5c1da37f41f7191ffd1e9d63342112b72d"}
Nov 03 23:53:04.848763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.842015    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7268566181f611a97e11f6e91469fc5c1da37f41f7191ffd1e9d63342112b72d"
Nov 03 23:53:04.848763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.842090    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"
Nov 03 23:53:04.848763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.848429    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb" event={"ID":"813afa38-726c-48a5-9b99-c47ac49db7ad","Type":"ContainerDied","Data":"1d56256e524dbf2dce2708d33887cf885b1725b8a0bebc82f9d27e0f88220937"}
Nov 03 23:53:04.848763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.848442    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"
Nov 03 23:53:04.848763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:04.848473    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d56256e524dbf2dce2708d33887cf885b1725b8a0bebc82f9d27e0f88220937"
Nov 03 23:53:13.315765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:13.285904    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 23:53:13.315765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:13.314619    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.477187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:13.476832    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.728437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:13.727773    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.014875 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.012864    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:53:14.329819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.328967    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded"
Nov 03 23:53:14.329819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.329259    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.329819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.329574    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 23:53:14.330400 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.329863    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:53:14.330400 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.329925    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:53:14.609811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:14.594881    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.074776 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.063828    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.423571 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.403945    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.485770 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.469472    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.729764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.729678    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:15.729764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.729754    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.730045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.729814    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.882161 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.882121    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:15.885835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.882417    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.929090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.929037    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:53:15.929337 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.929320    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:53:15.929466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.929454    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:53:15.929616 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.929603    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:53:15.929765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.929752    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:53:15.929920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:15.929899    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:53:19.542627 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.542589    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"]
Nov 03 23:53:19.543292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.543228    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa51b005-48b9-49d9-91a2-182dff56564a" containerName="revoke-keys"
Nov 03 23:53:19.543292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.543242    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa51b005-48b9-49d9-91a2-182dff56564a" containerName="revoke-keys"
Nov 03 23:53:19.543292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.543268    5074 cpu_manager.go:414] "containerMap: removing container" podUID="813afa38-726c-48a5-9b99-c47ac49db7ad" containerName="revoke-keys"
Nov 03 23:53:19.543292 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.543275    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813afa38-726c-48a5-9b99-c47ac49db7ad" containerName="revoke-keys"
Nov 03 23:53:19.543537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.543368    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="813afa38-726c-48a5-9b99-c47ac49db7ad" containerName="revoke-keys"
Nov 03 23:53:19.543537 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.543383    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa51b005-48b9-49d9-91a2-182dff56564a" containerName="revoke-keys"
Nov 03 23:53:19.547230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.547209    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.552415 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.552388    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:53:19.555057 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.555037    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:53:19.555794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.555766    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"]
Nov 03 23:53:19.690466 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.690432    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5kp\" (UniqueName: \"kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.690651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.690535    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.690735 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.690712    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.792063 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.792030    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5kp\" (UniqueName: \"kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.792242 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.792092    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.792242 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.792182    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.792985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.792921    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.794667 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.794640    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.805339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.805281    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5kp\" (UniqueName: \"kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp\") pod \"maas-api-revoke-sub-1018b3a0-46wvl\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:19.873312 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:19.873284    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:20.161482 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:20.161454    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"]
Nov 03 23:53:20.162062 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:20.162030    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode008ab4f_315c_49f6_bd78_a457887d86bb.slice/crio-24d46cd354e3ba0fd957a2d5958c1ecb6f247d388360a063c5e1366b38884a70 WatchSource:0}: Error finding container 24d46cd354e3ba0fd957a2d5958c1ecb6f247d388360a063c5e1366b38884a70: Status 404 returned error can't find the container with id 24d46cd354e3ba0fd957a2d5958c1ecb6f247d388360a063c5e1366b38884a70
Nov 03 23:53:20.944352 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:20.944319    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl" event={"ID":"e008ab4f-315c-49f6-bd78-a457887d86bb","Type":"ContainerStarted","Data":"750956ff08df662d28bc5c898a356ec38c11c46ec3e74d348535481fc48959c4"}
Nov 03 23:53:20.944784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:20.944379    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl" event={"ID":"e008ab4f-315c-49f6-bd78-a457887d86bb","Type":"ContainerStarted","Data":"24d46cd354e3ba0fd957a2d5958c1ecb6f247d388360a063c5e1366b38884a70"}
Nov 03 23:53:20.969307 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:20.969260    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl" podStartSLOduration=1.969247356 podStartE2EDuration="1.969247356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:20.965171391 +0000 UTC m=+1466.064771534" watchObservedRunningTime="2026-11-03 23:53:20.969247356 +0000 UTC m=+1466.068847535"
Nov 03 23:53:21.954337 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:21.954303    5074 generic.go:356] "Generic (PLEG): container finished" podID="e008ab4f-315c-49f6-bd78-a457887d86bb" containerID="750956ff08df662d28bc5c898a356ec38c11c46ec3e74d348535481fc48959c4" exitCode=0
Nov 03 23:53:21.954732 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:21.954370    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl" event={"ID":"e008ab4f-315c-49f6-bd78-a457887d86bb","Type":"ContainerDied","Data":"750956ff08df662d28bc5c898a356ec38c11c46ec3e74d348535481fc48959c4"}
Nov 03 23:53:23.155604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.155578    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:23.247308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.247217    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca\") pod \"e008ab4f-315c-49f6-bd78-a457887d86bb\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") "
Nov 03 23:53:23.247488 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.247307    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp\" (UniqueName: \"kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp\") pod \"e008ab4f-315c-49f6-bd78-a457887d86bb\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") "
Nov 03 23:53:23.247488 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.247412    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca\") pod \"e008ab4f-315c-49f6-bd78-a457887d86bb\" (UID: \"e008ab4f-315c-49f6-bd78-a457887d86bb\") "
Nov 03 23:53:23.247912 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.247881    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca" pod "e008ab4f-315c-49f6-bd78-a457887d86bb" (UID: "e008ab4f-315c-49f6-bd78-a457887d86bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:23.248059 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.248042    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:23.249732 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.249708    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca" pod "e008ab4f-315c-49f6-bd78-a457887d86bb" (UID: "e008ab4f-315c-49f6-bd78-a457887d86bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:23.249993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.249973    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp" pod "e008ab4f-315c-49f6-bd78-a457887d86bb" (UID: "e008ab4f-315c-49f6-bd78-a457887d86bb"). InnerVolumeSpecName "kube-api-access-lf5kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:23.355417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.355380    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5kp\" (UniqueName: \"kubernetes.io/projected/e008ab4f-315c-49f6-bd78-a457887d86bb-kube-api-access-lf5kp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:23.355852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.355833    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e008ab4f-315c-49f6-bd78-a457887d86bb-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:23.456881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.456826    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"]
Nov 03 23:53:23.458088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.458048    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e008ab4f-315c-49f6-bd78-a457887d86bb" containerName="revoke-keys"
Nov 03 23:53:23.458088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.458087    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e008ab4f-315c-49f6-bd78-a457887d86bb" containerName="revoke-keys"
Nov 03 23:53:23.458316 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.458302    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e008ab4f-315c-49f6-bd78-a457887d86bb" containerName="revoke-keys"
Nov 03 23:53:23.465133 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.465100    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.467202 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.467171    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"]
Nov 03 23:53:23.561450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.561342    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2d2\" (UniqueName: \"kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.561450 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.561393    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.561694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.561646    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.666164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.664844    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2d2\" (UniqueName: \"kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.666164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.665689    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.667842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.666601    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.667990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.667849    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.670318 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.670268    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.694033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.693988    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2d2\" (UniqueName: \"kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2\") pod \"maas-api-revoke-sub-7cac345a-krk89\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:23.787461 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:23.787426    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:24.027765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.019336    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"]
Nov 03 23:53:24.038810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.036032    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"
Nov 03 23:53:24.073868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.073826    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl" event={"ID":"e008ab4f-315c-49f6-bd78-a457887d86bb","Type":"ContainerDied","Data":"24d46cd354e3ba0fd957a2d5958c1ecb6f247d388360a063c5e1366b38884a70"}
Nov 03 23:53:24.076124 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.076103    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d46cd354e3ba0fd957a2d5958c1ecb6f247d388360a063c5e1366b38884a70"
Nov 03 23:53:24.076349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.076072    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.082239 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.082164    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"]
Nov 03 23:53:24.099296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.099206    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"]
Nov 03 23:53:24.114324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.113984    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxlh\" (UniqueName: \"kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.114324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.114050    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.114324 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.114153    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.118367 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:24.118346    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacd51a6_58d6_433a_945d_96408d93fcbe.slice/crio-6f370f787bf072e235817b0682cb456ed3f0ae122c68a7fa1f211dd012c3fcc5 WatchSource:0}: Error finding container 6f370f787bf072e235817b0682cb456ed3f0ae122c68a7fa1f211dd012c3fcc5: Status 404 returned error can't find the container with id 6f370f787bf072e235817b0682cb456ed3f0ae122c68a7fa1f211dd012c3fcc5
Nov 03 23:53:24.215854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.215823    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.216761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.216717    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.216910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.216891    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.217178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.217166    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxlh\" (UniqueName: \"kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.223607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.223584    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.231280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.231251    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxlh\" (UniqueName: \"kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh\") pod \"maas-api-revoke-sub-9122cf60-tg68g\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.412504 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.412422    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:24.710274 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:24.710232    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"]
Nov 03 23:53:24.712017 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:24.711812    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641a5425_f53b_4991_b168_d37ece30dad7.slice/crio-beeafe20d9f3d01c9a037969534fd1fcc7e16db526037b84c880c29c0b5285cb WatchSource:0}: Error finding container beeafe20d9f3d01c9a037969534fd1fcc7e16db526037b84c880c29c0b5285cb: Status 404 returned error can't find the container with id beeafe20d9f3d01c9a037969534fd1fcc7e16db526037b84c880c29c0b5285cb
Nov 03 23:53:25.173765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:25.173689    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g" event={"ID":"641a5425-f53b-4991-b168-d37ece30dad7","Type":"ContainerStarted","Data":"beeafe20d9f3d01c9a037969534fd1fcc7e16db526037b84c880c29c0b5285cb"}
Nov 03 23:53:25.176975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:25.176906    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89" event={"ID":"bacd51a6-58d6-433a-945d-96408d93fcbe","Type":"ContainerStarted","Data":"9d05b29da8ace1be533552abff6b9b8f37bada75c26aaf8ac8006101f1b976e6"}
Nov 03 23:53:25.176975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:25.176971    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89" event={"ID":"bacd51a6-58d6-433a-945d-96408d93fcbe","Type":"ContainerStarted","Data":"6f370f787bf072e235817b0682cb456ed3f0ae122c68a7fa1f211dd012c3fcc5"}
Nov 03 23:53:25.219906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:25.219849    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g" podStartSLOduration=2.219833436 podStartE2EDuration="2.219833436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:25.193573001 +0000 UTC m=+1470.293173146" watchObservedRunningTime="2026-11-03 23:53:25.219833436 +0000 UTC m=+1470.319433579"
Nov 03 23:53:26.279394 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:26.279356    5074 generic.go:356] "Generic (PLEG): container finished" podID="641a5425-f53b-4991-b168-d37ece30dad7" containerID="ef8ae58b087eaa99784c8e47a75518d9be7dee12155a822548b6714a0adc6924" exitCode=0
Nov 03 23:53:26.280102 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:26.280065    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g" event={"ID":"641a5425-f53b-4991-b168-d37ece30dad7","Type":"ContainerDied","Data":"ef8ae58b087eaa99784c8e47a75518d9be7dee12155a822548b6714a0adc6924"}
Nov 03 23:53:26.316763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:26.315697    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89" podStartSLOduration=3.3156787850000002 podStartE2EDuration="3.315678785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:25.220158736 +0000 UTC m=+1470.319758880" watchObservedRunningTime="2026-11-03 23:53:26.315678785 +0000 UTC m=+1471.415278930"
Nov 03 23:53:26.340819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:26.338057    5074 generic.go:356] "Generic (PLEG): container finished" podID="bacd51a6-58d6-433a-945d-96408d93fcbe" containerID="9d05b29da8ace1be533552abff6b9b8f37bada75c26aaf8ac8006101f1b976e6" exitCode=0
Nov 03 23:53:26.340819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:26.338164    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89" event={"ID":"bacd51a6-58d6-433a-945d-96408d93fcbe","Type":"ContainerDied","Data":"9d05b29da8ace1be533552abff6b9b8f37bada75c26aaf8ac8006101f1b976e6"}
Nov 03 23:53:28.081123 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:53:28.056049    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:53:28.131782 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.124663    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:28.134240 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.134212    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:28.223276 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.223232    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca\") pod \"641a5425-f53b-4991-b168-d37ece30dad7\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") "
Nov 03 23:53:28.223449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.223323    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca\") pod \"bacd51a6-58d6-433a-945d-96408d93fcbe\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") "
Nov 03 23:53:28.223449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.223356    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca\") pod \"641a5425-f53b-4991-b168-d37ece30dad7\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") "
Nov 03 23:53:28.223449 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.223415    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh\" (UniqueName: \"kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh\") pod \"641a5425-f53b-4991-b168-d37ece30dad7\" (UID: \"641a5425-f53b-4991-b168-d37ece30dad7\") "
Nov 03 23:53:28.223651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.223503    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2\" (UniqueName: \"kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2\") pod \"bacd51a6-58d6-433a-945d-96408d93fcbe\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") "
Nov 03 23:53:28.223651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.223610    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca\") pod \"bacd51a6-58d6-433a-945d-96408d93fcbe\" (UID: \"bacd51a6-58d6-433a-945d-96408d93fcbe\") "
Nov 03 23:53:28.224797 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.224764    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca" pod "bacd51a6-58d6-433a-945d-96408d93fcbe" (UID: "bacd51a6-58d6-433a-945d-96408d93fcbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:28.242193 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.242139    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:28.257389 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.257350    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca" pod "641a5425-f53b-4991-b168-d37ece30dad7" (UID: "641a5425-f53b-4991-b168-d37ece30dad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:28.274798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.274301    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2" pod "bacd51a6-58d6-433a-945d-96408d93fcbe" (UID: "bacd51a6-58d6-433a-945d-96408d93fcbe"). InnerVolumeSpecName "kube-api-access-6q2d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:28.274798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.274367    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh" pod "641a5425-f53b-4991-b168-d37ece30dad7" (UID: "641a5425-f53b-4991-b168-d37ece30dad7"). InnerVolumeSpecName "kube-api-access-vpxlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:28.282487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.277924    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca" pod "641a5425-f53b-4991-b168-d37ece30dad7" (UID: "641a5425-f53b-4991-b168-d37ece30dad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:28.282487 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.278241    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca" pod "bacd51a6-58d6-433a-945d-96408d93fcbe" (UID: "bacd51a6-58d6-433a-945d-96408d93fcbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:28.343545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.343467    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:28.343722 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.343704    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a5425-f53b-4991-b168-d37ece30dad7-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:28.343844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.343821    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxlh\" (UniqueName: \"kubernetes.io/projected/641a5425-f53b-4991-b168-d37ece30dad7-kube-api-access-vpxlh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:28.343946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.343846    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q2d2\" (UniqueName: \"kubernetes.io/projected/bacd51a6-58d6-433a-945d-96408d93fcbe-kube-api-access-6q2d2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:28.343946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.343862    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd51a6-58d6-433a-945d-96408d93fcbe-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:28.380957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.380929    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"
Nov 03 23:53:28.380957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.380943    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g" event={"ID":"641a5425-f53b-4991-b168-d37ece30dad7","Type":"ContainerDied","Data":"beeafe20d9f3d01c9a037969534fd1fcc7e16db526037b84c880c29c0b5285cb"}
Nov 03 23:53:28.381171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.381027    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beeafe20d9f3d01c9a037969534fd1fcc7e16db526037b84c880c29c0b5285cb"
Nov 03 23:53:28.382548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.382515    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89" event={"ID":"bacd51a6-58d6-433a-945d-96408d93fcbe","Type":"ContainerDied","Data":"6f370f787bf072e235817b0682cb456ed3f0ae122c68a7fa1f211dd012c3fcc5"}
Nov 03 23:53:28.382734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.382560    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f370f787bf072e235817b0682cb456ed3f0ae122c68a7fa1f211dd012c3fcc5"
Nov 03 23:53:28.382734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:28.382527    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"
Nov 03 23:53:29.769903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.769869    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"]
Nov 03 23:53:29.770802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.770778    5074 cpu_manager.go:414] "containerMap: removing container" podUID="641a5425-f53b-4991-b168-d37ece30dad7" containerName="revoke-keys"
Nov 03 23:53:29.770802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.770802    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641a5425-f53b-4991-b168-d37ece30dad7" containerName="revoke-keys"
Nov 03 23:53:29.770948 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.770836    5074 cpu_manager.go:414] "containerMap: removing container" podUID="bacd51a6-58d6-433a-945d-96408d93fcbe" containerName="revoke-keys"
Nov 03 23:53:29.770948 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.770841    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacd51a6-58d6-433a-945d-96408d93fcbe" containerName="revoke-keys"
Nov 03 23:53:29.771051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.770985    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacd51a6-58d6-433a-945d-96408d93fcbe" containerName="revoke-keys"
Nov 03 23:53:29.771051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.771002    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="641a5425-f53b-4991-b168-d37ece30dad7" containerName="revoke-keys"
Nov 03 23:53:29.774482 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.774460    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.781900 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.781876    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"]
Nov 03 23:53:29.782085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.782027    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:53:29.783643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.783620    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:53:29.865541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.865509    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.865712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.865627    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.865818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.865762    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pqc\" (UniqueName: \"kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.968110 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.968075    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.968295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.968242    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pqc\" (UniqueName: \"kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.968395 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.968378    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.969088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.969062    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.970600 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.970575    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:29.981035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:29.981014    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pqc\" (UniqueName: \"kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc\") pod \"maas-api-revoke-sub-072673a8-svpns\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:30.093929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.093831    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:30.495920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.495880    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"]
Nov 03 23:53:30.506800 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:30.503774    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf6c359_3890_493e_936d_35c4f79b7bba.slice/crio-bb3810398e80d57aebe3f1b89b73f7c823cfc57b928030ca4e0db7936fd10340 WatchSource:0}: Error finding container bb3810398e80d57aebe3f1b89b73f7c823cfc57b928030ca4e0db7936fd10340: Status 404 returned error can't find the container with id bb3810398e80d57aebe3f1b89b73f7c823cfc57b928030ca4e0db7936fd10340
Nov 03 23:53:30.513069 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.513023    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"]
Nov 03 23:53:30.527280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.521581    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.527280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.523202    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"]
Nov 03 23:53:30.588651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.588616    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.588932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.588911    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.589087 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589070    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.590006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589376    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.590006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589583    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.590006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589618    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.590006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589648    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.590006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589680    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.590006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.589711    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qvj\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.690749 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.690707    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.690962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.690799    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.690962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.690831    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.690962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.690861    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.690962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.690891    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qvj\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.691185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.690968    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.691185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.691036    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.691185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.691067    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.691185 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.691134    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.691375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.691185    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.691597 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.691573    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.692046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.692023    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.692326 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.692306    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.692496 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.692480    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.695130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.695103    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.695282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.695128    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.799477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.799391    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.799956 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.799475    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qvj\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:30.899171 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:30.899140    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:31.226824 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:31.218096    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823da97b_dcfd_4dbd_94fe_e6af3dbac78d.slice/crio-68d3b3738b547bc7a0c31f0f38a85cd81ba3b2037fcde17dfda7bed041b5964f WatchSource:0}: Error finding container 68d3b3738b547bc7a0c31f0f38a85cd81ba3b2037fcde17dfda7bed041b5964f: Status 404 returned error can't find the container with id 68d3b3738b547bc7a0c31f0f38a85cd81ba3b2037fcde17dfda7bed041b5964f
Nov 03 23:53:31.243767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.235936    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:53:31.243767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.236100    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:53:31.243767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.236155    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:53:31.244060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.243864    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"]
Nov 03 23:53:31.512778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.512720    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns" event={"ID":"bdf6c359-3890-493e-936d-35c4f79b7bba","Type":"ContainerStarted","Data":"ddac08a52153b80e74aae779559fe3b7e2bdcf8917e631128df20eba403ebb65"}
Nov 03 23:53:31.512959 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.512809    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns" event={"ID":"bdf6c359-3890-493e-936d-35c4f79b7bba","Type":"ContainerStarted","Data":"bb3810398e80d57aebe3f1b89b73f7c823cfc57b928030ca4e0db7936fd10340"}
Nov 03 23:53:31.515196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.515162    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" event={"ID":"823da97b-dcfd-4dbd-94fe-e6af3dbac78d","Type":"ContainerStarted","Data":"4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218"}
Nov 03 23:53:31.515329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.515221    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" event={"ID":"823da97b-dcfd-4dbd-94fe-e6af3dbac78d","Type":"ContainerStarted","Data":"68d3b3738b547bc7a0c31f0f38a85cd81ba3b2037fcde17dfda7bed041b5964f"}
Nov 03 23:53:31.538979 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.538930    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns" podStartSLOduration=2.538917919 podStartE2EDuration="2.538917919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:31.536501237 +0000 UTC m=+1476.636101385" watchObservedRunningTime="2026-11-03 23:53:31.538917919 +0000 UTC m=+1476.638518063"
Nov 03 23:53:31.568926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.568863    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podStartSLOduration=1.568844769 podStartE2EDuration="1.568844769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:31.562279098 +0000 UTC m=+1476.661879245" watchObservedRunningTime="2026-11-03 23:53:31.568844769 +0000 UTC m=+1476.668444914"
Nov 03 23:53:31.938255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:31.938161    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:32.546090 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:32.546045    5074 generic.go:356] "Generic (PLEG): container finished" podID="bdf6c359-3890-493e-936d-35c4f79b7bba" containerID="ddac08a52153b80e74aae779559fe3b7e2bdcf8917e631128df20eba403ebb65" exitCode=0
Nov 03 23:53:32.546293 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:32.546158    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns" event={"ID":"bdf6c359-3890-493e-936d-35c4f79b7bba","Type":"ContainerDied","Data":"ddac08a52153b80e74aae779559fe3b7e2bdcf8917e631128df20eba403ebb65"}
Nov 03 23:53:32.908272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:32.907840    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:53:32.908272 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:32.907942    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 23:53:33.873647 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.873620    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:33.926774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.922185    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:53:33.926774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.922278    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 23:53:33.975763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.975713    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca\") pod \"bdf6c359-3890-493e-936d-35c4f79b7bba\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") "
Nov 03 23:53:33.977056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.977033    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc\" (UniqueName: \"kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc\") pod \"bdf6c359-3890-493e-936d-35c4f79b7bba\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") "
Nov 03 23:53:33.977382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.976909    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca" pod "bdf6c359-3890-493e-936d-35c4f79b7bba" (UID: "bdf6c359-3890-493e-936d-35c4f79b7bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:33.980799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.978981    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca\") pod \"bdf6c359-3890-493e-936d-35c4f79b7bba\" (UID: \"bdf6c359-3890-493e-936d-35c4f79b7bba\") "
Nov 03 23:53:33.980799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.979660    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:33.981578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.981549    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca" pod "bdf6c359-3890-493e-936d-35c4f79b7bba" (UID: "bdf6c359-3890-493e-936d-35c4f79b7bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:33.982603 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:33.982578    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc" pod "bdf6c359-3890-493e-936d-35c4f79b7bba" (UID: "bdf6c359-3890-493e-936d-35c4f79b7bba"). InnerVolumeSpecName "kube-api-access-t5pqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:34.081604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.081447    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pqc\" (UniqueName: \"kubernetes.io/projected/bdf6c359-3890-493e-936d-35c4f79b7bba-kube-api-access-t5pqc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:34.081604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.081482    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf6c359-3890-493e-936d-35c4f79b7bba-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:34.572758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.572429    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"
Nov 03 23:53:34.572758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.572449    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns" event={"ID":"bdf6c359-3890-493e-936d-35c4f79b7bba","Type":"ContainerDied","Data":"bb3810398e80d57aebe3f1b89b73f7c823cfc57b928030ca4e0db7936fd10340"}
Nov 03 23:53:34.572758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.572511    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3810398e80d57aebe3f1b89b73f7c823cfc57b928030ca4e0db7936fd10340"
Nov 03 23:53:34.902811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.900167    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:53:34.902811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:34.900227    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 23:53:35.918762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:35.909084    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:53:35.940881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:35.940843    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 23:53:36.903764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:36.900374    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:53:36.903764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:36.900466    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 23:53:37.899957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.899905    5074 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:53:37.900483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.899987    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 23:53:37.941340 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.941305    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"]
Nov 03 23:53:37.942380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.942351    5074 cpu_manager.go:414] "containerMap: removing container" podUID="bdf6c359-3890-493e-936d-35c4f79b7bba" containerName="revoke-keys"
Nov 03 23:53:37.942380 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.942378    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf6c359-3890-493e-936d-35c4f79b7bba" containerName="revoke-keys"
Nov 03 23:53:37.942595 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.942518    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf6c359-3890-493e-936d-35c4f79b7bba" containerName="revoke-keys"
Nov 03 23:53:37.947064 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.947038    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:37.951993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.951960    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:53:37.952613 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.952588    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"]
Nov 03 23:53:37.953953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.953933    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:53:37.966430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.962948    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:37.966430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.963100    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:37.966430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:37.963135    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpnr\" (UniqueName: \"kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.064424 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.064384    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.064592 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.064432    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpnr\" (UniqueName: \"kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.064678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.064600    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.065391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.065362    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.067159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.067141    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.077649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.077620    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpnr\" (UniqueName: \"kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr\") pod \"maas-api-revoke-sub-6757a157-bgp64\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.283839 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.283808    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:38.592402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.592102    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"]
Nov 03 23:53:38.592515 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:38.592432    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783ff9cd_6a8a_4e96_9d1a_5ed038bdb892.slice/crio-2e1e7b57deeddec6182555e67f78eb12452df82e30e617359c90a2958ddf35d8 WatchSource:0}: Error finding container 2e1e7b57deeddec6182555e67f78eb12452df82e30e617359c90a2958ddf35d8: Status 404 returned error can't find the container with id 2e1e7b57deeddec6182555e67f78eb12452df82e30e617359c90a2958ddf35d8
Nov 03 23:53:38.609817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.609777    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64" event={"ID":"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892","Type":"ContainerStarted","Data":"2e1e7b57deeddec6182555e67f78eb12452df82e30e617359c90a2958ddf35d8"}
Nov 03 23:53:38.905862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.905775    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:38.906963 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.906800    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:38.908024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:38.907684    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 03 23:53:39.623398 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:39.623358    5074 generic.go:356] "Generic (PLEG): container finished" podID="783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" containerID="05917ecfb4b2bac518463b56ef9f04490980dd74d519cba00e0fb2adac765ce0" exitCode=0
Nov 03 23:53:39.623575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:39.623558    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64" event={"ID":"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892","Type":"ContainerDied","Data":"05917ecfb4b2bac518463b56ef9f04490980dd74d519cba00e0fb2adac765ce0"}
Nov 03 23:53:40.821323 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.821289    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:40.944043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.944008    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr\" (UniqueName: \"kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr\") pod \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") "
Nov 03 23:53:40.944223 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.944185    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca\") pod \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") "
Nov 03 23:53:40.944283 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.944265    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca\") pod \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\" (UID: \"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892\") "
Nov 03 23:53:40.944637 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.944611    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca" pod "783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" (UID: "783ff9cd-6a8a-4e96-9d1a-5ed038bdb892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:40.944908 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.944886    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.946688 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.946660    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca" pod "783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" (UID: "783ff9cd-6a8a-4e96-9d1a-5ed038bdb892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:40.946794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:40.946687    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr" pod "783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" (UID: "783ff9cd-6a8a-4e96-9d1a-5ed038bdb892"). InnerVolumeSpecName "kube-api-access-7kpnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:41.046708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:41.046678    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpnr\" (UniqueName: \"kubernetes.io/projected/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-kube-api-access-7kpnr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:41.046708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:41.046705    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:41.642036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:41.642005    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"
Nov 03 23:53:41.642224 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:41.642005    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64" event={"ID":"783ff9cd-6a8a-4e96-9d1a-5ed038bdb892","Type":"ContainerDied","Data":"2e1e7b57deeddec6182555e67f78eb12452df82e30e617359c90a2958ddf35d8"}
Nov 03 23:53:41.642224 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:41.642129    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1e7b57deeddec6182555e67f78eb12452df82e30e617359c90a2958ddf35d8"
Nov 03 23:53:43.428376 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:43.428322    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:43.962889 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:43.962844    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:43.999199 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:43.999160    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.320835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.302557    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.516761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.515896    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.516761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.516443    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:53:44.619846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.607317    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.742777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.734881    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.978764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.966912    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.978764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:44.969223    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:53:45.003372 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.002664    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:53:45.041769 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.009871    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:53:45.143814 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.143768    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.451818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.440031    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:45.451818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.440126    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.519783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.517023    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.608884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.608833    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.739813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.739694    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.821771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.820850    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:45.821771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.820960    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.969118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:45.969063    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.012798 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.012681    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.240437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.240391    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.280305 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.280208    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.301759 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:53:46.297618    5074 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.288s"
Nov 03 23:53:46.370508 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.370481    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:53:46.370821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.370802    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:53:46.372555 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.372536    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:53:46.520137 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.520076    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 23:53:46.639355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.639261    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.694919 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.694856    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.810764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.762665    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.966463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.966435    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:53:46.974008 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:46.973939    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:53:47.153308 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.153275    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:53:47.158520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.158496    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:53:47.159168 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.159150    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:53:47.185877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.185835    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:53:47.261785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.261703    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"]
Nov 03 23:53:47.263494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.263470    5074 cpu_manager.go:414] "containerMap: removing container" podUID="783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" containerName="revoke-keys"
Nov 03 23:53:47.263655 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.263640    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" containerName="revoke-keys"
Nov 03 23:53:47.263933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.263917    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" containerName="revoke-keys"
Nov 03 23:53:47.268051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.268029    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.277510 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.277399    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"]
Nov 03 23:53:47.293056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.293026    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:53:47.293191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.293153    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:53:47.372296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.372253    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.372973 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.372924    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.373161 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.373141    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcvl\" (UniqueName: \"kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.475038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.474800    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.475300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.475275    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcvl\" (UniqueName: \"kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.475542 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.475518    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.475660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.475599    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.478011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.477985    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.488297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.488277    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcvl\" (UniqueName: \"kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl\") pod \"maas-api-revoke-sub-2b4e3d9c-s6dhv\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.587761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.587671    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:47.886931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:47.886891    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"]
Nov 03 23:53:47.897084 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:53:47.897051    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdcda55_6aef_4977_9872_fbef4244524c.slice/crio-f31640c779640cbd9bab789c23b09a738f7cea4ddfa9d10995d95a8ac49b5335 WatchSource:0}: Error finding container f31640c779640cbd9bab789c23b09a738f7cea4ddfa9d10995d95a8ac49b5335: Status 404 returned error can't find the container with id f31640c779640cbd9bab789c23b09a738f7cea4ddfa9d10995d95a8ac49b5335
Nov 03 23:53:48.313429 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:48.313390    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv" event={"ID":"bbdcda55-6aef-4977-9872-fbef4244524c","Type":"ContainerStarted","Data":"f31640c779640cbd9bab789c23b09a738f7cea4ddfa9d10995d95a8ac49b5335"}
Nov 03 23:53:49.325656 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:49.325622    5074 generic.go:356] "Generic (PLEG): container finished" podID="bbdcda55-6aef-4977-9872-fbef4244524c" containerID="5283de9449ec696ba05f79c8acc356eff4dbfef17d603cfc2eb2f1bb4464cb89" exitCode=0
Nov 03 23:53:49.326183 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:49.325721    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv" event={"ID":"bbdcda55-6aef-4977-9872-fbef4244524c","Type":"ContainerDied","Data":"5283de9449ec696ba05f79c8acc356eff4dbfef17d603cfc2eb2f1bb4464cb89"}
Nov 03 23:53:50.509246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.509210    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:50.526483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.526452    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca\") pod \"bbdcda55-6aef-4977-9872-fbef4244524c\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") "
Nov 03 23:53:50.526811 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.526762    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca\") pod \"bbdcda55-6aef-4977-9872-fbef4244524c\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") "
Nov 03 23:53:50.526898 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.526838    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca" pod "bbdcda55-6aef-4977-9872-fbef4244524c" (UID: "bbdcda55-6aef-4977-9872-fbef4244524c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:50.527256 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.527233    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:50.530795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.530765    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca" pod "bbdcda55-6aef-4977-9872-fbef4244524c" (UID: "bbdcda55-6aef-4977-9872-fbef4244524c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:50.628835 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.628752    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl\" (UniqueName: \"kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl\") pod \"bbdcda55-6aef-4977-9872-fbef4244524c\" (UID: \"bbdcda55-6aef-4977-9872-fbef4244524c\") "
Nov 03 23:53:50.629078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.629066    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdcda55-6aef-4977-9872-fbef4244524c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:50.630904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.630874    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl" pod "bbdcda55-6aef-4977-9872-fbef4244524c" (UID: "bbdcda55-6aef-4977-9872-fbef4244524c"). InnerVolumeSpecName "kube-api-access-tgcvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:50.730469 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:50.730426    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgcvl\" (UniqueName: \"kubernetes.io/projected/bbdcda55-6aef-4977-9872-fbef4244524c-kube-api-access-tgcvl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:51.343375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:51.343290    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"
Nov 03 23:53:51.343375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:51.343339    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv" event={"ID":"bbdcda55-6aef-4977-9872-fbef4244524c","Type":"ContainerDied","Data":"f31640c779640cbd9bab789c23b09a738f7cea4ddfa9d10995d95a8ac49b5335"}
Nov 03 23:53:51.343598 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:51.343396    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31640c779640cbd9bab789c23b09a738f7cea4ddfa9d10995d95a8ac49b5335"
Nov 03 23:53:54.150490 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.150451    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"]
Nov 03 23:53:54.151382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.151363    5074 cpu_manager.go:414] "containerMap: removing container" podUID="bbdcda55-6aef-4977-9872-fbef4244524c" containerName="revoke-keys"
Nov 03 23:53:54.151428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.151383    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdcda55-6aef-4977-9872-fbef4244524c" containerName="revoke-keys"
Nov 03 23:53:54.151541 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.151530    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdcda55-6aef-4977-9872-fbef4244524c" containerName="revoke-keys"
Nov 03 23:53:54.156943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.156913    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.161660 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.161630    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"]
Nov 03 23:53:54.162841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.162803    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:53:54.164374 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.164357    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:53:54.278491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.278460    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.278491 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.278496    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9wx\" (UniqueName: \"kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.278790 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.278646    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.380669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.380637    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.380669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.380674    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9wx\" (UniqueName: \"kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.380914 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.380751    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.381300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.381279    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.383209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.383187    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.395388 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.395360    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9wx\" (UniqueName: \"kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx\") pod \"maas-api-revoke-sub-701cb87e-nf4rm\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.476865 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.476825    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:54.818430 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:54.818299    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"]
Nov 03 23:53:55.389676 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:55.389628    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm" event={"ID":"98b81f83-65db-4c81-a068-a35310a0f9dc","Type":"ContainerStarted","Data":"e4f07c9321e288481a5f0b111db768d9ffa89f3b423bd260ced9cae4d0eadb98"}
Nov 03 23:53:55.390306 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:55.389706    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm" event={"ID":"98b81f83-65db-4c81-a068-a35310a0f9dc","Type":"ContainerStarted","Data":"f735e8bbc94ab4b33950d9398ab1786466529b507a89007928ed5160983c6136"}
Nov 03 23:53:55.413028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:55.412965    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm" podStartSLOduration=1.412946015 podStartE2EDuration="1.412946015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:55.410844141 +0000 UTC m=+1500.510444287" watchObservedRunningTime="2026-11-03 23:53:55.412946015 +0000 UTC m=+1500.512546163"
Nov 03 23:53:55.415465 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:55.415428    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:53:55.465960 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:55.465871    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:53:56.270479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:56.270434    5074 scope.go:122] "RemoveContainer" containerID="04c37792efbafb8c7648cc1e09b62c7f7b91eca19689fe2e279a60e24a993b30"
Nov 03 23:53:56.285075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:56.285046    5074 scope.go:122] "RemoveContainer" containerID="d45f25336df05b94f2407f512efb469777812b267e21e6327c52f74e03e69e02"
Nov 03 23:53:56.297520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:56.297489    5074 scope.go:122] "RemoveContainer" containerID="d6346fb09ce7464b553bac60b6313aacb159159449d15c350f4d6cd959a927cf"
Nov 03 23:53:56.312968 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:56.312934    5074 scope.go:122] "RemoveContainer" containerID="1c1a5973177363609537bdf1cb7e09e0a8d6ffb9870bc521ed5c28a41ba7bba1"
Nov 03 23:53:56.400443 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:56.400407    5074 generic.go:356] "Generic (PLEG): container finished" podID="98b81f83-65db-4c81-a068-a35310a0f9dc" containerID="e4f07c9321e288481a5f0b111db768d9ffa89f3b423bd260ced9cae4d0eadb98" exitCode=0
Nov 03 23:53:56.401022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:56.400490    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm" event={"ID":"98b81f83-65db-4c81-a068-a35310a0f9dc","Type":"ContainerDied","Data":"e4f07c9321e288481a5f0b111db768d9ffa89f3b423bd260ced9cae4d0eadb98"}
Nov 03 23:53:57.601756 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.601718    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:57.736627 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.736595    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx\" (UniqueName: \"kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx\") pod \"98b81f83-65db-4c81-a068-a35310a0f9dc\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") "
Nov 03 23:53:57.736846 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.736711    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca\") pod \"98b81f83-65db-4c81-a068-a35310a0f9dc\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") "
Nov 03 23:53:57.736918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.736862    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca\") pod \"98b81f83-65db-4c81-a068-a35310a0f9dc\" (UID: \"98b81f83-65db-4c81-a068-a35310a0f9dc\") "
Nov 03 23:53:57.737168 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.737140    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca" pod "98b81f83-65db-4c81-a068-a35310a0f9dc" (UID: "98b81f83-65db-4c81-a068-a35310a0f9dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:57.737388 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.737370    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:57.739351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.739321    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca" pod "98b81f83-65db-4c81-a068-a35310a0f9dc" (UID: "98b81f83-65db-4c81-a068-a35310a0f9dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:57.739459 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.739367    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx" pod "98b81f83-65db-4c81-a068-a35310a0f9dc" (UID: "98b81f83-65db-4c81-a068-a35310a0f9dc"). InnerVolumeSpecName "kube-api-access-4f9wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:57.838200 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.838166    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9wx\" (UniqueName: \"kubernetes.io/projected/98b81f83-65db-4c81-a068-a35310a0f9dc-kube-api-access-4f9wx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:57.838200 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:57.838193    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b81f83-65db-4c81-a068-a35310a0f9dc-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:58.031487 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:53:58.031368    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:53:58.419151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:58.419063    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"
Nov 03 23:53:58.419151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:58.419088    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm" event={"ID":"98b81f83-65db-4c81-a068-a35310a0f9dc","Type":"ContainerDied","Data":"f735e8bbc94ab4b33950d9398ab1786466529b507a89007928ed5160983c6136"}
Nov 03 23:53:58.419151 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:58.419137    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f735e8bbc94ab4b33950d9398ab1786466529b507a89007928ed5160983c6136"
Nov 03 23:53:59.278085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.278045    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"]
Nov 03 23:53:59.279384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.279356    5074 cpu_manager.go:414] "containerMap: removing container" podUID="98b81f83-65db-4c81-a068-a35310a0f9dc" containerName="revoke-keys"
Nov 03 23:53:59.279384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.279382    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b81f83-65db-4c81-a068-a35310a0f9dc" containerName="revoke-keys"
Nov 03 23:53:59.279727 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.279709    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b81f83-65db-4c81-a068-a35310a0f9dc" containerName="revoke-keys"
Nov 03 23:53:59.350996 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.350965    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"]
Nov 03 23:53:59.353695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.351324    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.372939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.371292    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:53:59.372939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.372785    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:53:59.378327 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.377600    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"]
Nov 03 23:53:59.418092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.406319    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"]
Nov 03 23:53:59.418092 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.406455    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.473943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.473728    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.474176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.474151    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.475039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.475006    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khfq\" (UniqueName: \"kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.475182 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.475090    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzml\" (UniqueName: \"kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.475258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.475197    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.475332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.475310    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.576818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.576698    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzml\" (UniqueName: \"kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.576818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.576792    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.577011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.576848    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.577011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.576938    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.577011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.576980    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.577176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.577022    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khfq\" (UniqueName: \"kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.578481 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.578447    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.578783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.578754    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.580499 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.580475    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.580659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.580445    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.590036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.590008    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khfq\" (UniqueName: \"kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq\") pod \"maas-api-revoke-sub-c7a59128-zjqjv\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.590217 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.590195    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzml\" (UniqueName: \"kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml\") pod \"maas-api-revoke-sub-c2c65eb1-8sbm5\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:53:59.770107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.770070    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:53:59.774024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:53:59.773994    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:54:00.065015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:00.064986    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"]
Nov 03 23:54:00.065348 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:00.065330    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178d8d46_053a_4ca3_8bb0_01c6d0a7c8c0.slice/crio-ec29d66860b0308906c9d388164d0494159ba1ea5dffb8b606f341be1882658b WatchSource:0}: Error finding container ec29d66860b0308906c9d388164d0494159ba1ea5dffb8b606f341be1882658b: Status 404 returned error can't find the container with id ec29d66860b0308906c9d388164d0494159ba1ea5dffb8b606f341be1882658b
Nov 03 23:54:00.078607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:00.078580    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"]
Nov 03 23:54:00.081043 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:00.081012    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aa0b5b_f72f_4d96_a2c6_1be9dae50553.slice/crio-4bec15bd853b7e31fea36c48b180fba9c5780913aaa390359a3326fd7cfa9962 WatchSource:0}: Error finding container 4bec15bd853b7e31fea36c48b180fba9c5780913aaa390359a3326fd7cfa9962: Status 404 returned error can't find the container with id 4bec15bd853b7e31fea36c48b180fba9c5780913aaa390359a3326fd7cfa9962
Nov 03 23:54:00.457483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:00.457411    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv" event={"ID":"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0","Type":"ContainerStarted","Data":"ec29d66860b0308906c9d388164d0494159ba1ea5dffb8b606f341be1882658b"}
Nov 03 23:54:00.459299 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:00.459268    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5" event={"ID":"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553","Type":"ContainerStarted","Data":"4bec15bd853b7e31fea36c48b180fba9c5780913aaa390359a3326fd7cfa9962"}
Nov 03 23:54:01.468552 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:01.468518    5074 generic.go:356] "Generic (PLEG): container finished" podID="178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" containerID="5264b6660ce40b8c7d0dedf730c57eaeb20dd879a4f026b04a14d2b15646e7a4" exitCode=0
Nov 03 23:54:01.468995 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:01.468605    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv" event={"ID":"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0","Type":"ContainerDied","Data":"5264b6660ce40b8c7d0dedf730c57eaeb20dd879a4f026b04a14d2b15646e7a4"}
Nov 03 23:54:01.470335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:01.470311    5074 generic.go:356] "Generic (PLEG): container finished" podID="d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" containerID="99f110a19b87cd667b32cfb5bab50098ff6b5e70477b204a76045d930df8a31e" exitCode=0
Nov 03 23:54:01.470485 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:01.470351    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5" event={"ID":"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553","Type":"ContainerDied","Data":"99f110a19b87cd667b32cfb5bab50098ff6b5e70477b204a76045d930df8a31e"}
Nov 03 23:54:02.841560 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.841214    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:54:02.931593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.930711    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:54:02.941424 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.941399    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml\" (UniqueName: \"kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml\") pod \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") "
Nov 03 23:54:02.941716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.941703    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca\") pod \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") "
Nov 03 23:54:02.941964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.941946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca\") pod \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\" (UID: \"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553\") "
Nov 03 23:54:02.942859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.942359    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca" pod "d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" (UID: "d5aa0b5b-f72f-4d96-a2c6-1be9dae50553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:02.942859 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.942816    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:02.956904 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.944703    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca" pod "d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" (UID: "d5aa0b5b-f72f-4d96-a2c6-1be9dae50553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:02.993976 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:02.991979    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml" pod "d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" (UID: "d5aa0b5b-f72f-4d96-a2c6-1be9dae50553"). InnerVolumeSpecName "kube-api-access-xpzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:03.043570 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.043538    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca\") pod \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") "
Nov 03 23:54:03.043816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.043789    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq\" (UniqueName: \"kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq\") pod \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") "
Nov 03 23:54:03.043978 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.043962    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca\") pod \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\" (UID: \"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0\") "
Nov 03 23:54:03.044352 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.044328    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca" pod "178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" (UID: "178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:03.044580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.044566    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:03.044634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.044583    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:03.044634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.044593    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzml\" (UniqueName: \"kubernetes.io/projected/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553-kube-api-access-xpzml\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:03.045651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.045628    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca" pod "178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" (UID: "178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:03.045790 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.045770    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq" pod "178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" (UID: "178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0"). InnerVolumeSpecName "kube-api-access-7khfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:03.146309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.146271    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:03.146309 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.146308    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khfq\" (UniqueName: \"kubernetes.io/projected/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0-kube-api-access-7khfq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:03.498626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.498587    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv" event={"ID":"178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0","Type":"ContainerDied","Data":"ec29d66860b0308906c9d388164d0494159ba1ea5dffb8b606f341be1882658b"}
Nov 03 23:54:03.499447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.499410    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec29d66860b0308906c9d388164d0494159ba1ea5dffb8b606f341be1882658b"
Nov 03 23:54:03.499607 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.498942    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"
Nov 03 23:54:03.502777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.502030    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5" event={"ID":"d5aa0b5b-f72f-4d96-a2c6-1be9dae50553","Type":"ContainerDied","Data":"4bec15bd853b7e31fea36c48b180fba9c5780913aaa390359a3326fd7cfa9962"}
Nov 03 23:54:03.502777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.502090    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bec15bd853b7e31fea36c48b180fba9c5780913aaa390359a3326fd7cfa9962"
Nov 03 23:54:03.502777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:03.502157    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"
Nov 03 23:54:16.398232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.398187    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"]
Nov 03 23:54:16.399939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.399909    5074 cpu_manager.go:414] "containerMap: removing container" podUID="178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" containerName="revoke-keys"
Nov 03 23:54:16.399939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.399937    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" containerName="revoke-keys"
Nov 03 23:54:16.400136 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.400017    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" containerName="revoke-keys"
Nov 03 23:54:16.400136 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.400027    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" containerName="revoke-keys"
Nov 03 23:54:16.400327 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.400309    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" containerName="revoke-keys"
Nov 03 23:54:16.400383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.400342    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" containerName="revoke-keys"
Nov 03 23:54:16.408468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.408441    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.421009 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.420974    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:54:16.421207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.421147    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:54:16.423291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.423266    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"]
Nov 03 23:54:16.562079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.562034    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.562279 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.562139    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxk8v\" (UniqueName: \"kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.562431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.562413    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.666794 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.666676    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.667120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.667092    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.667319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.667299    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxk8v\" (UniqueName: \"kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.668886 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.668864    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.685464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.685435    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxk8v\" (UniqueName: \"kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.688758 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.686315    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a09800-d8fpb\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:16.762557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:16.762522    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:17.130762 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:17.130654    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d9c722_06ab_4521_9f5e_c8716dd96e69.slice/crio-1129b09ba5784f234376f84dad867cffc1b5e1e22a4e3d6c24152938cb78be5f WatchSource:0}: Error finding container 1129b09ba5784f234376f84dad867cffc1b5e1e22a4e3d6c24152938cb78be5f: Status 404 returned error can't find the container with id 1129b09ba5784f234376f84dad867cffc1b5e1e22a4e3d6c24152938cb78be5f
Nov 03 23:54:17.134785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.131113    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"]
Nov 03 23:54:17.432116 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.432039    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"]
Nov 03 23:54:17.436431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.436411    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.444699 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.444664    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"]
Nov 03 23:54:17.615564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.615516    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.615753 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.615712    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.615813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.615787    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtpkx\" (UniqueName: \"kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.717578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.717544    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.717800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.717695    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.717900 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.717825    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtpkx\" (UniqueName: \"kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.718410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.718380    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.721787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.721118    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.733075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.733052    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtpkx\" (UniqueName: \"kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx\") pod \"maas-api-revoke-sub-587ef888-sds9s\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.754857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.754827    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:17.763705 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.763677    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb" event={"ID":"86d9c722-06ab-4521-9f5e-c8716dd96e69","Type":"ContainerStarted","Data":"2eaf4124de543d94e06f63137f7437aabef3615165468e513de39f9e4ac6fe74"}
Nov 03 23:54:17.763819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.763732    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb" event={"ID":"86d9c722-06ab-4521-9f5e-c8716dd96e69","Type":"ContainerStarted","Data":"1129b09ba5784f234376f84dad867cffc1b5e1e22a4e3d6c24152938cb78be5f"}
Nov 03 23:54:17.787900 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:17.787853    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb" podStartSLOduration=1.787837347 podStartE2EDuration="1.787837347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:17.786932886 +0000 UTC m=+1522.886533030" watchObservedRunningTime="2026-11-03 23:54:17.787837347 +0000 UTC m=+1522.887437490"
Nov 03 23:54:18.057966 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:18.057937    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb115faa0_6316_4e24_9a24_d693e5711d1a.slice/crio-cfb7b374fd95701d5bf9314fae9f9b94de848f634dee9152ab5bb1a2c2311e70 WatchSource:0}: Error finding container cfb7b374fd95701d5bf9314fae9f9b94de848f634dee9152ab5bb1a2c2311e70: Status 404 returned error can't find the container with id cfb7b374fd95701d5bf9314fae9f9b94de848f634dee9152ab5bb1a2c2311e70
Nov 03 23:54:18.059411 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:18.059389    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"]
Nov 03 23:54:18.772158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:18.772122    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s" event={"ID":"b115faa0-6316-4e24-9a24-d693e5711d1a","Type":"ContainerStarted","Data":"f2a4849e5f6623c86bef04a59bacb7263ec9781c274f8d2c888646f58fdd17a3"}
Nov 03 23:54:18.772615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:18.772186    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s" event={"ID":"b115faa0-6316-4e24-9a24-d693e5711d1a","Type":"ContainerStarted","Data":"cfb7b374fd95701d5bf9314fae9f9b94de848f634dee9152ab5bb1a2c2311e70"}
Nov 03 23:54:18.773787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:18.773757    5074 generic.go:356] "Generic (PLEG): container finished" podID="86d9c722-06ab-4521-9f5e-c8716dd96e69" containerID="2eaf4124de543d94e06f63137f7437aabef3615165468e513de39f9e4ac6fe74" exitCode=0
Nov 03 23:54:18.773894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:18.773824    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb" event={"ID":"86d9c722-06ab-4521-9f5e-c8716dd96e69","Type":"ContainerDied","Data":"2eaf4124de543d94e06f63137f7437aabef3615165468e513de39f9e4ac6fe74"}
Nov 03 23:54:18.808298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:18.807275    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s" podStartSLOduration=1.807254937 podStartE2EDuration="1.807254937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:18.796138287 +0000 UTC m=+1523.895738434" watchObservedRunningTime="2026-11-03 23:54:18.807254937 +0000 UTC m=+1523.906855082"
Nov 03 23:54:19.782096 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:19.782061    5074 generic.go:356] "Generic (PLEG): container finished" podID="b115faa0-6316-4e24-9a24-d693e5711d1a" containerID="f2a4849e5f6623c86bef04a59bacb7263ec9781c274f8d2c888646f58fdd17a3" exitCode=0
Nov 03 23:54:19.782527 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:19.782142    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s" event={"ID":"b115faa0-6316-4e24-9a24-d693e5711d1a","Type":"ContainerDied","Data":"f2a4849e5f6623c86bef04a59bacb7263ec9781c274f8d2c888646f58fdd17a3"}
Nov 03 23:54:19.960857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:19.960831    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:20.149777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.149658    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca\") pod \"86d9c722-06ab-4521-9f5e-c8716dd96e69\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") "
Nov 03 23:54:20.149777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.149704    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v\" (UniqueName: \"kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v\") pod \"86d9c722-06ab-4521-9f5e-c8716dd96e69\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") "
Nov 03 23:54:20.150029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.149877    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca\") pod \"86d9c722-06ab-4521-9f5e-c8716dd96e69\" (UID: \"86d9c722-06ab-4521-9f5e-c8716dd96e69\") "
Nov 03 23:54:20.150336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.150309    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca" pod "86d9c722-06ab-4521-9f5e-c8716dd96e69" (UID: "86d9c722-06ab-4521-9f5e-c8716dd96e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:20.150515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.150496    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:20.151920 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.151883    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca" pod "86d9c722-06ab-4521-9f5e-c8716dd96e69" (UID: "86d9c722-06ab-4521-9f5e-c8716dd96e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:20.152038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.152021    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v" pod "86d9c722-06ab-4521-9f5e-c8716dd96e69" (UID: "86d9c722-06ab-4521-9f5e-c8716dd96e69"). InnerVolumeSpecName "kube-api-access-lxk8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:20.252400 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.252360    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9c722-06ab-4521-9f5e-c8716dd96e69-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:20.252400 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.252396    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxk8v\" (UniqueName: \"kubernetes.io/projected/86d9c722-06ab-4521-9f5e-c8716dd96e69-kube-api-access-lxk8v\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:20.792030 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.792000    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"
Nov 03 23:54:20.792463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.792036    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb" event={"ID":"86d9c722-06ab-4521-9f5e-c8716dd96e69","Type":"ContainerDied","Data":"1129b09ba5784f234376f84dad867cffc1b5e1e22a4e3d6c24152938cb78be5f"}
Nov 03 23:54:20.792463 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.792096    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1129b09ba5784f234376f84dad867cffc1b5e1e22a4e3d6c24152938cb78be5f"
Nov 03 23:54:20.983946 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:20.983913    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:21.164024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.163942    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx\" (UniqueName: \"kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx\") pod \"b115faa0-6316-4e24-9a24-d693e5711d1a\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") "
Nov 03 23:54:21.164196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.164031    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca\") pod \"b115faa0-6316-4e24-9a24-d693e5711d1a\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") "
Nov 03 23:54:21.164196 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.164101    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca\") pod \"b115faa0-6316-4e24-9a24-d693e5711d1a\" (UID: \"b115faa0-6316-4e24-9a24-d693e5711d1a\") "
Nov 03 23:54:21.164385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.164354    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca" pod "b115faa0-6316-4e24-9a24-d693e5711d1a" (UID: "b115faa0-6316-4e24-9a24-d693e5711d1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:21.164775 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.164735    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:21.166199 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.166177    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca" pod "b115faa0-6316-4e24-9a24-d693e5711d1a" (UID: "b115faa0-6316-4e24-9a24-d693e5711d1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:21.166199 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.166190    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx" pod "b115faa0-6316-4e24-9a24-d693e5711d1a" (UID: "b115faa0-6316-4e24-9a24-d693e5711d1a"). InnerVolumeSpecName "kube-api-access-gtpkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:21.266241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.266205    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b115faa0-6316-4e24-9a24-d693e5711d1a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:21.266241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.266233    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtpkx\" (UniqueName: \"kubernetes.io/projected/b115faa0-6316-4e24-9a24-d693e5711d1a-kube-api-access-gtpkx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:21.800521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.800489    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"
Nov 03 23:54:21.800521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.800513    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s" event={"ID":"b115faa0-6316-4e24-9a24-d693e5711d1a","Type":"ContainerDied","Data":"cfb7b374fd95701d5bf9314fae9f9b94de848f634dee9152ab5bb1a2c2311e70"}
Nov 03 23:54:21.800964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:21.800569    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb7b374fd95701d5bf9314fae9f9b94de848f634dee9152ab5bb1a2c2311e70"
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.847341    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"]
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.848315    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b115faa0-6316-4e24-9a24-d693e5711d1a" containerName="revoke-keys"
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.848337    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b115faa0-6316-4e24-9a24-d693e5711d1a" containerName="revoke-keys"
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.848389    5074 cpu_manager.go:414] "containerMap: removing container" podUID="86d9c722-06ab-4521-9f5e-c8716dd96e69" containerName="revoke-keys"
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.848397    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d9c722-06ab-4521-9f5e-c8716dd96e69" containerName="revoke-keys"
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.848556    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b115faa0-6316-4e24-9a24-d693e5711d1a" containerName="revoke-keys"
Nov 03 23:54:26.876765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.848579    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d9c722-06ab-4521-9f5e-c8716dd96e69" containerName="revoke-keys"
Nov 03 23:54:26.906819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.891401    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:26.947756 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.944282    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"]
Nov 03 23:54:26.990678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.990636    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.021763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:26.997728    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.037336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.023136    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:54:27.037336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.023210    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:54:27.037336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.029672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.037336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.029754    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22v7\" (UniqueName: \"kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.037336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.030014    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.105807 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.100926    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.170840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.170639    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.170840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.170699    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22v7\" (UniqueName: \"kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.192778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.192106    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.192778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.192395    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.364972 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.364890    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.373683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.373648    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22v7\" (UniqueName: \"kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7\") pod \"maas-api-revoke-sub-d3e27c18-6mt4g\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.475763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.471985    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:27.818985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.818525    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:54:27.818985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:27.818603    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:28.006797 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:27.999119    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55541821_33a8_438f_9e36_85b78645aeaf.slice/crio-e42cbbc637cd9dbbdcb1ab29629f040701b2665fda37d3d6a02ad2ac30161e7b WatchSource:0}: Error finding container e42cbbc637cd9dbbdcb1ab29629f040701b2665fda37d3d6a02ad2ac30161e7b: Status 404 returned error can't find the container with id e42cbbc637cd9dbbdcb1ab29629f040701b2665fda37d3d6a02ad2ac30161e7b
Nov 03 23:54:28.009037 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:28.008382    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"]
Nov 03 23:54:28.047964 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:54:28.043183    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:54:28.180792 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:28.180755    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g" event={"ID":"55541821-33a8-438f-9e36-85b78645aeaf","Type":"ContainerStarted","Data":"e42cbbc637cd9dbbdcb1ab29629f040701b2665fda37d3d6a02ad2ac30161e7b"}
Nov 03 23:54:29.190093 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:29.190055    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g" event={"ID":"55541821-33a8-438f-9e36-85b78645aeaf","Type":"ContainerStarted","Data":"6c05217cd657cfa78c2a50edcf3c07fd10bf2ead218ba158cb056bd0f3f177fe"}
Nov 03 23:54:29.214970 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:29.214920    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g" podStartSLOduration=3.214906253 podStartE2EDuration="3.214906253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:29.209330382 +0000 UTC m=+1534.308930527" watchObservedRunningTime="2026-11-03 23:54:29.214906253 +0000 UTC m=+1534.314506397"
Nov 03 23:54:30.198771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:30.198723    5074 generic.go:356] "Generic (PLEG): container finished" podID="55541821-33a8-438f-9e36-85b78645aeaf" containerID="6c05217cd657cfa78c2a50edcf3c07fd10bf2ead218ba158cb056bd0f3f177fe" exitCode=0
Nov 03 23:54:30.199241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:30.198822    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g" event={"ID":"55541821-33a8-438f-9e36-85b78645aeaf","Type":"ContainerDied","Data":"6c05217cd657cfa78c2a50edcf3c07fd10bf2ead218ba158cb056bd0f3f177fe"}
Nov 03 23:54:31.627201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.627172    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:31.754777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.754726    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca\") pod \"55541821-33a8-438f-9e36-85b78645aeaf\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") "
Nov 03 23:54:31.754970 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.754806    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca\") pod \"55541821-33a8-438f-9e36-85b78645aeaf\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") "
Nov 03 23:54:31.755040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.754965    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7\" (UniqueName: \"kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7\") pod \"55541821-33a8-438f-9e36-85b78645aeaf\" (UID: \"55541821-33a8-438f-9e36-85b78645aeaf\") "
Nov 03 23:54:31.755187 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.755164    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca" pod "55541821-33a8-438f-9e36-85b78645aeaf" (UID: "55541821-33a8-438f-9e36-85b78645aeaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:31.755525 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.755507    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:31.756922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.756899    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca" pod "55541821-33a8-438f-9e36-85b78645aeaf" (UID: "55541821-33a8-438f-9e36-85b78645aeaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:31.757033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.757016    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7" pod "55541821-33a8-438f-9e36-85b78645aeaf" (UID: "55541821-33a8-438f-9e36-85b78645aeaf"). InnerVolumeSpecName "kube-api-access-h22v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:31.856933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.856847    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22v7\" (UniqueName: \"kubernetes.io/projected/55541821-33a8-438f-9e36-85b78645aeaf-kube-api-access-h22v7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:31.856933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:31.856879    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55541821-33a8-438f-9e36-85b78645aeaf-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:32.257046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:32.257015    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"
Nov 03 23:54:32.257227 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:32.257023    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g" event={"ID":"55541821-33a8-438f-9e36-85b78645aeaf","Type":"ContainerDied","Data":"e42cbbc637cd9dbbdcb1ab29629f040701b2665fda37d3d6a02ad2ac30161e7b"}
Nov 03 23:54:32.257227 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:32.257140    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42cbbc637cd9dbbdcb1ab29629f040701b2665fda37d3d6a02ad2ac30161e7b"
Nov 03 23:54:36.076390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.076354    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"]
Nov 03 23:54:36.077074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.077058    5074 cpu_manager.go:414] "containerMap: removing container" podUID="55541821-33a8-438f-9e36-85b78645aeaf" containerName="revoke-keys"
Nov 03 23:54:36.077134 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.077075    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55541821-33a8-438f-9e36-85b78645aeaf" containerName="revoke-keys"
Nov 03 23:54:36.077189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.077178    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="55541821-33a8-438f-9e36-85b78645aeaf" containerName="revoke-keys"
Nov 03 23:54:36.080761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.080729    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.086099 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.086075    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:54:36.092885 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.089209    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"]
Nov 03 23:54:36.112082 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.112057    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:54:36.132569 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.132542    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrcz\" (UniqueName: \"kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.132865 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.132836    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.132982 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.132940    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.234076 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.234043    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrcz\" (UniqueName: \"kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.234248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.234098    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.234248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.234150    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.234841 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.234818    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.236764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.236717    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.246926 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.246895    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrcz\" (UniqueName: \"kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz\") pod \"maas-api-revoke-sub-a2f044c2-zbx7v\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.418557 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.418477    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:36.708156 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:36.708119    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6dae55_c36d_49c1_bfbb_3545b4e28bb9.slice/crio-4fb52621ff235079d12f2ce99b65c256c3ba921f7084b9c719151680b089ee43 WatchSource:0}: Error finding container 4fb52621ff235079d12f2ce99b65c256c3ba921f7084b9c719151680b089ee43: Status 404 returned error can't find the container with id 4fb52621ff235079d12f2ce99b65c256c3ba921f7084b9c719151680b089ee43
Nov 03 23:54:36.711477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:36.710733    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"]
Nov 03 23:54:37.334362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:37.334256    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v" event={"ID":"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9","Type":"ContainerStarted","Data":"236c67f8dba143ecf5e619f035533b2a1eb9d1e6a8b8a592f2ca1e8efe814730"}
Nov 03 23:54:37.334362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:37.334350    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v" event={"ID":"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9","Type":"ContainerStarted","Data":"4fb52621ff235079d12f2ce99b65c256c3ba921f7084b9c719151680b089ee43"}
Nov 03 23:54:37.374122 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:37.374058    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v" podStartSLOduration=1.374042768 podStartE2EDuration="1.374042768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:37.372577722 +0000 UTC m=+1542.472177878" watchObservedRunningTime="2026-11-03 23:54:37.374042768 +0000 UTC m=+1542.473642957"
Nov 03 23:54:38.343682 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:38.343646    5074 generic.go:356] "Generic (PLEG): container finished" podID="bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" containerID="236c67f8dba143ecf5e619f035533b2a1eb9d1e6a8b8a592f2ca1e8efe814730" exitCode=0
Nov 03 23:54:38.344159 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:38.343774    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v" event={"ID":"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9","Type":"ContainerDied","Data":"236c67f8dba143ecf5e619f035533b2a1eb9d1e6a8b8a592f2ca1e8efe814730"}
Nov 03 23:54:39.667484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.667459    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:39.699877 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.699840    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz\" (UniqueName: \"kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz\") pod \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") "
Nov 03 23:54:39.700046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.699941    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca\") pod \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") "
Nov 03 23:54:39.703695 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.703664    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz" pod "bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" (UID: "bd6dae55-c36d-49c1-bfbb-3545b4e28bb9"). InnerVolumeSpecName "kube-api-access-2qrcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:39.704489 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.703887    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca\") pod \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\" (UID: \"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9\") "
Nov 03 23:54:39.705190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.705161    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qrcz\" (UniqueName: \"kubernetes.io/projected/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-kube-api-access-2qrcz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:39.711967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.711935    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca" pod "bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" (UID: "bd6dae55-c36d-49c1-bfbb-3545b4e28bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:39.713176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.713148    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca" pod "bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" (UID: "bd6dae55-c36d-49c1-bfbb-3545b4e28bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:39.807212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.807181    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:39.807212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:39.807212    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:40.367692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:40.367660    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"
Nov 03 23:54:40.367692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:40.367677    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v" event={"ID":"bd6dae55-c36d-49c1-bfbb-3545b4e28bb9","Type":"ContainerDied","Data":"4fb52621ff235079d12f2ce99b65c256c3ba921f7084b9c719151680b089ee43"}
Nov 03 23:54:40.367921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:40.367726    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb52621ff235079d12f2ce99b65c256c3ba921f7084b9c719151680b089ee43"
Nov 03 23:54:49.976892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:49.976821    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.013921 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.013857    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.118947 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.118904    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.532873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.532826    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.644006 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.643957    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.841965 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.841855    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.027764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.979825    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 23:54:51.027764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:50.980292    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:54:51.035716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.028538    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.035716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.029340    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:54:51.167821 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.151870    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:54:51.413404 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.413373    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:54:51.538275 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.538228    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.539336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.539280    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:54:51.663259 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.663195    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.705216 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:51.705176    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:54:52.170870 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.170821    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.194191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.193644    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.194480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.194449    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.203984 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.194790    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.283568 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.283529    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Nov 03 23:54:52.295764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.295723    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:54:52.326768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.325823    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 03 23:54:52.583638 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.583602    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:54:52.766802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.718334    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 03 23:54:52.766802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.722085    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 03 23:54:52.783050 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.783022    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:54:52.805124 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:52.788041    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 03 23:54:55.868943 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.868910    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"]
Nov 03 23:54:55.869817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.869794    5074 cpu_manager.go:414] "containerMap: removing container" podUID="bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" containerName="revoke-keys"
Nov 03 23:54:55.869817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.869816    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" containerName="revoke-keys"
Nov 03 23:54:55.870012 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.869971    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" containerName="revoke-keys"
Nov 03 23:54:55.874177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.874155    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:55.885358 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.885331    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"]
Nov 03 23:54:55.887026 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.887004    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:54:55.887439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.887403    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:54:55.943712 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.943669    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:55.943962 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.943931    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:55.944033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:55.943965    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9gs\" (UniqueName: \"kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.045297 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.045260    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.045503 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.045398    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.045622 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.045597    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9gs\" (UniqueName: \"kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.046120 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.046087    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.048246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.048221    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.061053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.061026    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9gs\" (UniqueName: \"kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs\") pod \"maas-api-revoke-sub-9c12e230-k9s9s\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.194107 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.194069    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:56.369887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.369843    5074 scope.go:122] "RemoveContainer" containerID="11f21774d20e2880f8b4e45b47879414a65f76c956dfd1456f38bb6f147bef60"
Nov 03 23:54:56.497172 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:56.497142    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"]
Nov 03 23:54:56.497509 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:54:56.497477    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f1ed1c_668e_4ee4_96c7_022057322e96.slice/crio-6aa424eba2d6cb55ce3244ecd9e1f0c516652be5d9193467949dd11ae1172330 WatchSource:0}: Error finding container 6aa424eba2d6cb55ce3244ecd9e1f0c516652be5d9193467949dd11ae1172330: Status 404 returned error can't find the container with id 6aa424eba2d6cb55ce3244ecd9e1f0c516652be5d9193467949dd11ae1172330
Nov 03 23:54:57.459207 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:57.459162    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s" event={"ID":"49f1ed1c-668e-4ee4-96c7-022057322e96","Type":"ContainerStarted","Data":"3aa5950f71ee29820cd61e579d2be31673b8dcdba03e05f8854341a0b0f5712f"}
Nov 03 23:54:57.459726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:57.459229    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s" event={"ID":"49f1ed1c-668e-4ee4-96c7-022057322e96","Type":"ContainerStarted","Data":"6aa424eba2d6cb55ce3244ecd9e1f0c516652be5d9193467949dd11ae1172330"}
Nov 03 23:54:57.482444 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:57.482390    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s" podStartSLOduration=2.482372767 podStartE2EDuration="2.482372767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:57.478134736 +0000 UTC m=+1562.577734880" watchObservedRunningTime="2026-11-03 23:54:57.482372767 +0000 UTC m=+1562.581972911"
Nov 03 23:54:58.023432 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:54:58.023395    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:54:58.481795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:58.481756    5074 generic.go:356] "Generic (PLEG): container finished" podID="49f1ed1c-668e-4ee4-96c7-022057322e96" containerID="3aa5950f71ee29820cd61e579d2be31673b8dcdba03e05f8854341a0b0f5712f" exitCode=0
Nov 03 23:54:58.482298 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:58.481822    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s" event={"ID":"49f1ed1c-668e-4ee4-96c7-022057322e96","Type":"ContainerDied","Data":"3aa5950f71ee29820cd61e579d2be31673b8dcdba03e05f8854341a0b0f5712f"}
Nov 03 23:54:59.681652 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.681622    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:54:59.804028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.803945    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca\") pod \"49f1ed1c-668e-4ee4-96c7-022057322e96\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") "
Nov 03 23:54:59.804028 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.803998    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs\" (UniqueName: \"kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs\") pod \"49f1ed1c-668e-4ee4-96c7-022057322e96\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") "
Nov 03 23:54:59.804235 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.804132    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca\") pod \"49f1ed1c-668e-4ee4-96c7-022057322e96\" (UID: \"49f1ed1c-668e-4ee4-96c7-022057322e96\") "
Nov 03 23:54:59.804383 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.804356    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca" pod "49f1ed1c-668e-4ee4-96c7-022057322e96" (UID: "49f1ed1c-668e-4ee4-96c7-022057322e96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:59.804608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.804593    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:59.806135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.806111    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs" pod "49f1ed1c-668e-4ee4-96c7-022057322e96" (UID: "49f1ed1c-668e-4ee4-96c7-022057322e96"). InnerVolumeSpecName "kube-api-access-sl9gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:59.806249 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.806170    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca" pod "49f1ed1c-668e-4ee4-96c7-022057322e96" (UID: "49f1ed1c-668e-4ee4-96c7-022057322e96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:59.906350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.906312    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f1ed1c-668e-4ee4-96c7-022057322e96-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:59.906350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:54:59.906346    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9gs\" (UniqueName: \"kubernetes.io/projected/49f1ed1c-668e-4ee4-96c7-022057322e96-kube-api-access-sl9gs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:00.327175 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.327135    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"]
Nov 03 23:55:00.328173 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.328152    5074 cpu_manager.go:414] "containerMap: removing container" podUID="49f1ed1c-668e-4ee4-96c7-022057322e96" containerName="revoke-keys"
Nov 03 23:55:00.328232 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.328178    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f1ed1c-668e-4ee4-96c7-022057322e96" containerName="revoke-keys"
Nov 03 23:55:00.328358 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.328345    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f1ed1c-668e-4ee4-96c7-022057322e96" containerName="revoke-keys"
Nov 03 23:55:00.332752 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.332712    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.337952 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.337926    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"]
Nov 03 23:55:00.413716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.413684    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.413902 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.413765    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.413967 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.413959    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.501562 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.501533    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"
Nov 03 23:55:00.501771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.501557    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s" event={"ID":"49f1ed1c-668e-4ee4-96c7-022057322e96","Type":"ContainerDied","Data":"6aa424eba2d6cb55ce3244ecd9e1f0c516652be5d9193467949dd11ae1172330"}
Nov 03 23:55:00.501771 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.501620    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aa424eba2d6cb55ce3244ecd9e1f0c516652be5d9193467949dd11ae1172330"
Nov 03 23:55:00.515121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.515093    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.515310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.515294    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.515413 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.515398    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.516135 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.516108    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.517894 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.517869    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.530462 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.530435    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2\") pod \"maas-api-revoke-sub-f10a7c73-gh9rt\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.652723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.652640    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:00.941574 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:55:00.941540    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23deb7ce_d111_40ce_805a_fa5bf2095905.slice/crio-4ee821e9da4b8a6040803b92459d5ac816a0ded9618137e34cda8df3b7ec3042 WatchSource:0}: Error finding container 4ee821e9da4b8a6040803b92459d5ac816a0ded9618137e34cda8df3b7ec3042: Status 404 returned error can't find the container with id 4ee821e9da4b8a6040803b92459d5ac816a0ded9618137e34cda8df3b7ec3042
Nov 03 23:55:00.944713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:00.944592    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"]
Nov 03 23:55:01.514211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:01.514175    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt" event={"ID":"23deb7ce-d111-40ce-805a-fa5bf2095905","Type":"ContainerStarted","Data":"5c1ade97bef61d4d317bf230550209dc2ec1c3ab2160d5f4ebb77d5b4cef9fd8"}
Nov 03 23:55:01.514418 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:01.514238    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt" event={"ID":"23deb7ce-d111-40ce-805a-fa5bf2095905","Type":"ContainerStarted","Data":"4ee821e9da4b8a6040803b92459d5ac816a0ded9618137e34cda8df3b7ec3042"}
Nov 03 23:55:01.549716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:01.549661    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt" podStartSLOduration=1.549648544 podStartE2EDuration="1.549648544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:01.544716884 +0000 UTC m=+1566.644317027" watchObservedRunningTime="2026-11-03 23:55:01.549648544 +0000 UTC m=+1566.649248682"
Nov 03 23:55:02.534903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:02.534862    5074 generic.go:356] "Generic (PLEG): container finished" podID="23deb7ce-d111-40ce-805a-fa5bf2095905" containerID="5c1ade97bef61d4d317bf230550209dc2ec1c3ab2160d5f4ebb77d5b4cef9fd8" exitCode=0
Nov 03 23:55:02.535405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:02.534965    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt" event={"ID":"23deb7ce-d111-40ce-805a-fa5bf2095905","Type":"ContainerDied","Data":"5c1ade97bef61d4d317bf230550209dc2ec1c3ab2160d5f4ebb77d5b4cef9fd8"}
Nov 03 23:55:03.759917 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.759889    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:03.879386 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.879303    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca\") pod \"23deb7ce-d111-40ce-805a-fa5bf2095905\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") "
Nov 03 23:55:03.879540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.879514    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca\") pod \"23deb7ce-d111-40ce-805a-fa5bf2095905\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") "
Nov 03 23:55:03.879578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.879544    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2\") pod \"23deb7ce-d111-40ce-805a-fa5bf2095905\" (UID: \"23deb7ce-d111-40ce-805a-fa5bf2095905\") "
Nov 03 23:55:03.879682 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.879657    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca" pod "23deb7ce-d111-40ce-805a-fa5bf2095905" (UID: "23deb7ce-d111-40ce-805a-fa5bf2095905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:55:03.880106 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.880087    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:03.881878 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.881848    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca" pod "23deb7ce-d111-40ce-805a-fa5bf2095905" (UID: "23deb7ce-d111-40ce-805a-fa5bf2095905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:55:03.881981 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.881886    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2" pod "23deb7ce-d111-40ce-805a-fa5bf2095905" (UID: "23deb7ce-d111-40ce-805a-fa5bf2095905"). InnerVolumeSpecName "kube-api-access-nlfm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:55:03.981131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.981094    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23deb7ce-d111-40ce-805a-fa5bf2095905-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:03.981131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:03.981131    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/23deb7ce-d111-40ce-805a-fa5bf2095905-kube-api-access-nlfm2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:04.552578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:04.552543    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt" event={"ID":"23deb7ce-d111-40ce-805a-fa5bf2095905","Type":"ContainerDied","Data":"4ee821e9da4b8a6040803b92459d5ac816a0ded9618137e34cda8df3b7ec3042"}
Nov 03 23:55:04.552578 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:04.552573    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"
Nov 03 23:55:04.552832 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:04.552594    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ee821e9da4b8a6040803b92459d5ac816a0ded9618137e34cda8df3b7ec3042"
Nov 03 23:55:05.693030 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.692950    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"]
Nov 03 23:55:05.693575 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.693560    5074 cpu_manager.go:414] "containerMap: removing container" podUID="23deb7ce-d111-40ce-805a-fa5bf2095905" containerName="revoke-keys"
Nov 03 23:55:05.693629 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.693575    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23deb7ce-d111-40ce-805a-fa5bf2095905" containerName="revoke-keys"
Nov 03 23:55:05.693709 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.693699    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="23deb7ce-d111-40ce-805a-fa5bf2095905" containerName="revoke-keys"
Nov 03 23:55:05.697520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.697502    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.702047 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.702018    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:55:05.702708 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.702685    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:55:05.728767 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.725191    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"]
Nov 03 23:55:05.831258 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.831214    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8j54\" (UniqueName: \"kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.831502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.831484    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.831585 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.831530    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.933502 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.933458    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.933711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.933528    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.933711 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.933621    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8j54\" (UniqueName: \"kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.934673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.934640    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.936382 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.936350    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:05.946907 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:05.946846    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8j54\" (UniqueName: \"kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54\") pod \"maas-api-revoke-sub-9e83219f-4nh6l\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:06.049866 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:06.049834    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:06.379310 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:55:06.379271    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22afadfe_91ed_4263_a27d_9e0b342496da.slice/crio-e1479ee5615306ff0beb01894bda038f613a43659ebf7d9f9648faff20d0693f WatchSource:0}: Error finding container e1479ee5615306ff0beb01894bda038f613a43659ebf7d9f9648faff20d0693f: Status 404 returned error can't find the container with id e1479ee5615306ff0beb01894bda038f613a43659ebf7d9f9648faff20d0693f
Nov 03 23:55:06.379914 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:06.379810    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"]
Nov 03 23:55:06.567827 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:06.567784    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l" event={"ID":"22afadfe-91ed-4263-a27d-9e0b342496da","Type":"ContainerStarted","Data":"e1479ee5615306ff0beb01894bda038f613a43659ebf7d9f9648faff20d0693f"}
Nov 03 23:55:07.581873 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:07.581831    5074 generic.go:356] "Generic (PLEG): container finished" podID="22afadfe-91ed-4263-a27d-9e0b342496da" containerID="08f4eed46c99673b4a002653aea7c3b4cb5966f5b63e26ebfd609a4f9a746f86" exitCode=0
Nov 03 23:55:07.582420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:07.581950    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l" event={"ID":"22afadfe-91ed-4263-a27d-9e0b342496da","Type":"ContainerDied","Data":"08f4eed46c99673b4a002653aea7c3b4cb5966f5b63e26ebfd609a4f9a746f86"}
Nov 03 23:55:08.809304 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.809273    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:08.891788 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.891664    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54\" (UniqueName: \"kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54\") pod \"22afadfe-91ed-4263-a27d-9e0b342496da\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") "
Nov 03 23:55:08.891964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.891864    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca\") pod \"22afadfe-91ed-4263-a27d-9e0b342496da\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") "
Nov 03 23:55:08.891964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.891936    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca\") pod \"22afadfe-91ed-4263-a27d-9e0b342496da\" (UID: \"22afadfe-91ed-4263-a27d-9e0b342496da\") "
Nov 03 23:55:08.892263 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.892238    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca" pod "22afadfe-91ed-4263-a27d-9e0b342496da" (UID: "22afadfe-91ed-4263-a27d-9e0b342496da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:55:08.892456 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.892429    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:08.893945 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.893922    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54" pod "22afadfe-91ed-4263-a27d-9e0b342496da" (UID: "22afadfe-91ed-4263-a27d-9e0b342496da"). InnerVolumeSpecName "kube-api-access-v8j54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:55:08.894024 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.894001    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca" pod "22afadfe-91ed-4263-a27d-9e0b342496da" (UID: "22afadfe-91ed-4263-a27d-9e0b342496da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:55:08.994209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.994158    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8j54\" (UniqueName: \"kubernetes.io/projected/22afadfe-91ed-4263-a27d-9e0b342496da-kube-api-access-v8j54\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:08.994209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:08.994218    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22afadfe-91ed-4263-a27d-9e0b342496da-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:09.616840 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:09.616791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l" event={"ID":"22afadfe-91ed-4263-a27d-9e0b342496da","Type":"ContainerDied","Data":"e1479ee5615306ff0beb01894bda038f613a43659ebf7d9f9648faff20d0693f"}
Nov 03 23:55:09.617053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:09.616914    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1479ee5615306ff0beb01894bda038f613a43659ebf7d9f9648faff20d0693f"
Nov 03 23:55:09.617117 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:09.617057    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"
Nov 03 23:55:12.294762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:12.293834    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.394514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:12.394468    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:55:12.709624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:12.709578    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.724530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:12.724485    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 23:55:12.772910 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:12.772863    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.813881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:12.813833    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.162820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.160985    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:55:13.162820 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.161070    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.573787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.571900    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:55:13.573787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.571983    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.650931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.650882    5074 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:55:13.651242 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.651194    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.767294 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.745903    5074 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:55:13.767294 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.745980    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.825903 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.796171    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 23:55:13.828464 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.827195    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.848175 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.848108    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:55:13.848682 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:13.848610    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:55:14.317764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:14.314336    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:14.446543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:14.446515    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 03 23:55:14.501119 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:14.474839    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 23:55:14.610509 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:14.610417    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 03 23:55:27.558795 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.558541    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"]
Nov 03 23:55:27.560701 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.560666    5074 cpu_manager.go:414] "containerMap: removing container" podUID="22afadfe-91ed-4263-a27d-9e0b342496da" containerName="revoke-keys"
Nov 03 23:55:27.561609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.561144    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22afadfe-91ed-4263-a27d-9e0b342496da" containerName="revoke-keys"
Nov 03 23:55:27.562079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.562044    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22afadfe-91ed-4263-a27d-9e0b342496da" containerName="revoke-keys"
Nov 03 23:55:27.568172 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.568150    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.576359 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.576331    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"]
Nov 03 23:55:27.586913 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.586890    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2341ca\""
Nov 03 23:55:27.594887 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.594862    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"]
Nov 03 23:55:27.600145 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.600124    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.626267 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.626238    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2341ca-dockercfg-6zkrc\""
Nov 03 23:55:27.640928 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.640892    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"]
Nov 03 23:55:27.662458 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.662425    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.662713 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.662689    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7sbz\" (UniqueName: \"kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz\") pod \"payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.764091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.764045    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7d8\" (UniqueName: \"kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8\") pod \"payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.764527 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.764506    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.764690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.764677    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7sbz\" (UniqueName: \"kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz\") pod \"payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.764816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.764800    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.769137 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.765889    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.787241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.787144    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7sbz\" (UniqueName: \"kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz\") pod \"payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.868818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.866673    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.868818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.866866    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7d8\" (UniqueName: \"kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8\") pod \"payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.868818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.868307    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.893633 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.893172    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7d8\" (UniqueName: \"kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8\") pod \"payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:27.896385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.896153    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:27.966789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:27.966581    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:28.056637 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:55:28.056146    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:55:28.246787 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:28.244683    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"]
Nov 03 23:55:28.362039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:28.360841    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"]
Nov 03 23:55:29.046634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.046595    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb" event={"ID":"295a683e-40ef-4e48-85b1-25f3d1603f52","Type":"ContainerStarted","Data":"6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5"}
Nov 03 23:55:29.047248 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.047222    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb" event={"ID":"295a683e-40ef-4e48-85b1-25f3d1603f52","Type":"ContainerStarted","Data":"bcfa13b8d8fd83a2ae9c39cce43aa4df05fb617f710c1de76a3370a663ce77cf"}
Nov 03 23:55:29.047352 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.047268    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:29.049110 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.049083    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t" event={"ID":"737b7b5e-1188-4b76-9cea-fc60e57d4165","Type":"ContainerStarted","Data":"2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f"}
Nov 03 23:55:29.049240 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.049134    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t" event={"ID":"737b7b5e-1188-4b76-9cea-fc60e57d4165","Type":"ContainerStarted","Data":"21da4f3cc10dbed2023c90aaa428b8eca94cf14bcb62954192ad0d9298c8d04e"}
Nov 03 23:55:29.049312 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.049257    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:29.073170 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.073096    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb" podStartSLOduration=2.073076552 podStartE2EDuration="2.073076552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:29.067679629 +0000 UTC m=+1594.167279775" watchObservedRunningTime="2026-11-03 23:55:29.073076552 +0000 UTC m=+1594.172676697"
Nov 03 23:55:29.099805 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:29.099087    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t" podStartSLOduration=2.09906882 podStartE2EDuration="2.09906882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:29.090356454 +0000 UTC m=+1594.189956602" watchObservedRunningTime="2026-11-03 23:55:29.09906882 +0000 UTC m=+1594.198668967"
Nov 03 23:55:31.443934 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.443889    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"]
Nov 03 23:55:31.451649 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.451612    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.455319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.455294    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"]
Nov 03 23:55:31.461039 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.461010    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"]
Nov 03 23:55:31.462094 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.462068    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9a5948-dockercfg-htzb2\""
Nov 03 23:55:31.462206 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.462189    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9a5948\""
Nov 03 23:55:31.468391 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.468364    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.487489 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.487468    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"]
Nov 03 23:55:31.522558 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.522520    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.522760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.522650    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpb2g\" (UniqueName: \"kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g\") pod \"payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.623960 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.623919    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.624165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.624008    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpb2g\" (UniqueName: \"kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g\") pod \"payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.624269 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.624252    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhpj\" (UniqueName: \"kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj\") pod \"payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.624332 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.624303    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.625074 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.625044    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.641622 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.641583    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpb2g\" (UniqueName: \"kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g\") pod \"payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.735540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.730109    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhpj\" (UniqueName: \"kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj\") pod \"payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.735540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.730228    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.735540 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.731976    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.748813 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.748779    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhpj\" (UniqueName: \"kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj\") pod \"payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:31.784277 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.784224    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:31.795084 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:31.795054    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:32.161723 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:32.161691    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"]
Nov 03 23:55:32.214091 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:55:32.214063    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fea0052_1521_4874_82f8_bcf7b425172f.slice/crio-58c1f61f9fa96f77071c9140db54bbd3e87c6cfa3ae6aa0569172cebb6a566cd WatchSource:0}: Error finding container 58c1f61f9fa96f77071c9140db54bbd3e87c6cfa3ae6aa0569172cebb6a566cd: Status 404 returned error can't find the container with id 58c1f61f9fa96f77071c9140db54bbd3e87c6cfa3ae6aa0569172cebb6a566cd
Nov 03 23:55:32.224819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:32.223637    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"]
Nov 03 23:55:33.103764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.099806    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" event={"ID":"6fea0052-1521-4874-82f8-bcf7b425172f","Type":"ContainerStarted","Data":"32ab51e9f2d163ff4256c5757e492279ee171674a97780bd6406b6802d45a2ab"}
Nov 03 23:55:33.103764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.099885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" event={"ID":"6fea0052-1521-4874-82f8-bcf7b425172f","Type":"ContainerStarted","Data":"58c1f61f9fa96f77071c9140db54bbd3e87c6cfa3ae6aa0569172cebb6a566cd"}
Nov 03 23:55:33.103764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.099916    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:33.103764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.101797    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" event={"ID":"e546e4c8-8ed6-41d2-b44d-ede826ff75d1","Type":"ContainerStarted","Data":"ad2076d4450aee90eae70f5f0d2578f886b26615c5ba846b26bc5905349cfd01"}
Nov 03 23:55:33.103764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.101846    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" event={"ID":"e546e4c8-8ed6-41d2-b44d-ede826ff75d1","Type":"ContainerStarted","Data":"436c6a52b20cd540f68acace2c3b4ee9c01ca9d4fd4f6e4cf7e1c55658d70e52"}
Nov 03 23:55:33.103764 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.101878    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:33.124426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.124374    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" podStartSLOduration=2.124360945 podStartE2EDuration="2.124360945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:33.123644695 +0000 UTC m=+1598.223244839" watchObservedRunningTime="2026-11-03 23:55:33.124360945 +0000 UTC m=+1598.223961090"
Nov 03 23:55:33.179763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.176385    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" podStartSLOduration=2.176360376 podStartE2EDuration="2.176360376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:33.164137358 +0000 UTC m=+1598.263737502" watchObservedRunningTime="2026-11-03 23:55:33.176360376 +0000 UTC m=+1598.275960524"
Nov 03 23:55:33.331256 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.331196    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"]
Nov 03 23:55:33.385678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.385593    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"]
Nov 03 23:55:33.385862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.385698    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.396595 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.393100    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-b3bea7\""
Nov 03 23:55:33.448432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.448392    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.448619 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.448547    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lg4v\" (UniqueName: \"kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v\") pod \"payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.549953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.549900    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.550157 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.550010    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lg4v\" (UniqueName: \"kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v\") pod \"payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.550661 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.550642    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.563784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.563718    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lg4v\" (UniqueName: \"kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v\") pod \"payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:33.719760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:33.718984    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:34.809765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:34.802407    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"]
Nov 03 23:55:35.292378 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:35.292338    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" event={"ID":"4626a9ed-a96d-4640-b657-8334384dc755","Type":"ContainerStarted","Data":"382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6"}
Nov 03 23:55:35.292577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:35.292390    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" event={"ID":"4626a9ed-a96d-4640-b657-8334384dc755","Type":"ContainerStarted","Data":"f5cd4c3e165f5b8bbca5af31c8965c03b58ecb23ae9a197185faff62a1aa9c87"}
Nov 03 23:55:35.292577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:35.292441    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:35.313319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:35.313268    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" podStartSLOduration=2.31325397 podStartE2EDuration="2.31325397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:35.31145207 +0000 UTC m=+1600.411052215" watchObservedRunningTime="2026-11-03 23:55:35.31325397 +0000 UTC m=+1600.412854127"
Nov 03 23:55:39.282697 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:39.282660    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"]
Nov 03 23:55:39.283251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:39.282875    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy" containerID="cri-o://1be59122d087c7c323b98610e43e59337a27cbf773a2fc20169d9a4aac19fa3d" gracePeriod=30
Nov 03 23:55:40.062512 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:40.062480    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 03 23:55:40.062854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:40.062833    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 03 23:55:43.487331 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.487290    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"]
Nov 03 23:55:43.493726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.493695    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.500718 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.500682    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"]
Nov 03 23:55:43.501772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.501703    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8093f5-dockercfg-t2mt8\""
Nov 03 23:55:43.502483 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.502454    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8093f5\""
Nov 03 23:55:43.517856 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.517796    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"]
Nov 03 23:55:43.558264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.558229    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.585392 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.585319    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"]
Nov 03 23:55:43.598236 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.598196    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdnwv\" (UniqueName: \"kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv\") pod \"payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.598417 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.598314    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.702022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.700625    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.702022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.700702    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.702022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.700988    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdnwv\" (UniqueName: \"kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv\") pod \"payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.702022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.701025    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhckq\" (UniqueName: \"kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq\") pod \"payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.703573 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.703230    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.761018 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.750488    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdnwv\" (UniqueName: \"kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv\") pod \"payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.837806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.805894    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhckq\" (UniqueName: \"kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq\") pod \"payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.837806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.806059    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.837806 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.827568    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.865678 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.865087    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:43.869215 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.869155    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhckq\" (UniqueName: \"kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq\") pod \"payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:43.892351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:43.891086    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:44.237257 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:55:44.237224    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e16c7d_f400_4b5a_9af4_6365060d9576.slice/crio-379efbb57e611419b4a72568cde79feb52f9cf394c6493edbc51be54b9285491 WatchSource:0}: Error finding container 379efbb57e611419b4a72568cde79feb52f9cf394c6493edbc51be54b9285491: Status 404 returned error can't find the container with id 379efbb57e611419b4a72568cde79feb52f9cf394c6493edbc51be54b9285491
Nov 03 23:55:44.238315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.238272    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"]
Nov 03 23:55:44.245438 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.245301    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"]
Nov 03 23:55:44.288721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.286916    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 03 23:55:44.302482 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.302422    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 03 23:55:44.408097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.408050    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" event={"ID":"d6def27d-ec93-4920-a8c6-0044521b9a03","Type":"ContainerStarted","Data":"544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f"}
Nov 03 23:55:44.408241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.408131    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" event={"ID":"d6def27d-ec93-4920-a8c6-0044521b9a03","Type":"ContainerStarted","Data":"a83f64a772998a4d5c1c3ebf829c595c8420f864ccb6df66d6e255bf831b1002"}
Nov 03 23:55:44.408241 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.408165    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:44.410534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.410500    5074 generic.go:356] "Generic (PLEG): container finished" podID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerID="1be59122d087c7c323b98610e43e59337a27cbf773a2fc20169d9a4aac19fa3d" exitCode=0
Nov 03 23:55:44.410694 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.410671    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" event={"ID":"013b9e70-4fe3-478f-b31b-0138f2d2dd7b","Type":"ContainerDied","Data":"1be59122d087c7c323b98610e43e59337a27cbf773a2fc20169d9a4aac19fa3d"}
Nov 03 23:55:44.412863 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.412832    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" event={"ID":"d7e16c7d-f400-4b5a-9af4-6365060d9576","Type":"ContainerStarted","Data":"ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915"}
Nov 03 23:55:44.412957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.412886    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" event={"ID":"d7e16c7d-f400-4b5a-9af4-6365060d9576","Type":"ContainerStarted","Data":"379efbb57e611419b4a72568cde79feb52f9cf394c6493edbc51be54b9285491"}
Nov 03 23:55:44.432925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.432868    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" podStartSLOduration=1.4328537049999999 podStartE2EDuration="1.432853705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:44.429484312 +0000 UTC m=+1609.529084456" watchObservedRunningTime="2026-11-03 23:55:44.432853705 +0000 UTC m=+1609.532453849"
Nov 03 23:55:44.453836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.453784    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" podStartSLOduration=1.453765969 podStartE2EDuration="1.453765969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:44.449426835 +0000 UTC m=+1609.549026978" watchObservedRunningTime="2026-11-03 23:55:44.453765969 +0000 UTC m=+1609.553366118"
Nov 03 23:55:44.553433 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.553408    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741205    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741306    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741458    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741508    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741545    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741604    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741630    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741661    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.744836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.741703    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy\") pod \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\" (UID: \"013b9e70-4fe3-478f-b31b-0138f2d2dd7b\") "
Nov 03 23:55:44.748857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.746537    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:55:44.748857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.746721    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:55:44.748857 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.747035    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:55:44.754089 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.753731    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:55:44.757088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.757045    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:55:44.757833 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.757341    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:55:44.774230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.774190    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "kube-api-access-wgfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:55:44.774230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.774208    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:55:44.774612 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.774586    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo" pod "013b9e70-4fe3-478f-b31b-0138f2d2dd7b" (UID: "013b9e70-4fe3-478f-b31b-0138f2d2dd7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.849952    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.849991    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgfdb\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-kube-api-access-wgfdb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850012    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850025    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850039    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850052    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850065    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850079    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:44.851867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:44.850093    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b9e70-4fe3-478f-b31b-0138f2d2dd7b-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:55:45.422998 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:45.422962    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"
Nov 03 23:55:45.423209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:45.422989    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j" event={"ID":"013b9e70-4fe3-478f-b31b-0138f2d2dd7b","Type":"ContainerDied","Data":"7bff755ac0bfffbdade387a1c25dfb0291331f097f14edc59979f18cf50590c2"}
Nov 03 23:55:45.423209 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:45.423066    5074 scope.go:122] "RemoveContainer" containerID="1be59122d087c7c323b98610e43e59337a27cbf773a2fc20169d9a4aac19fa3d"
Nov 03 23:55:45.423406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:45.423281    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:45.451291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:45.451093    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"]
Nov 03 23:55:45.458043 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:45.458014    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-bv54j"]
Nov 03 23:55:46.302698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:46.302667    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 03 23:55:47.022477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:47.022445    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" path="/var/lib/kubelet/pods/013b9e70-4fe3-478f-b31b-0138f2d2dd7b/volumes"
Nov 03 23:55:51.864044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.864009    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"]
Nov 03 23:55:51.885204 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.885168    5074 cpu_manager.go:414] "containerMap: removing container" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy"
Nov 03 23:55:51.885204 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.885203    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy"
Nov 03 23:55:51.885460 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.885445    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="013b9e70-4fe3-478f-b31b-0138f2d2dd7b" containerName="istio-proxy"
Nov 03 23:55:51.890111 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.890089    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:51.895300 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.895273    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"]
Nov 03 23:55:51.897246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.897224    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"kube-root-ca.crt\""
Nov 03 23:55:51.897375 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.897360    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"default-dockercfg-qz7cn\""
Nov 03 23:55:51.897455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.897364    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"openshift-service-ca.crt\""
Nov 03 23:55:51.897455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.897436    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"auth-test-model-4331a7-kserve-self-signed-certs\""
Nov 03 23:55:51.968914 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.968867    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:51.969105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.969001    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:51.969105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.969096    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:51.969192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.969121    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmg7l\" (UniqueName: \"kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:51.969251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.969225    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:51.969384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:51.969366    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.081786 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.070449    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.082563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.082441    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.082810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.082614    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.083153 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.083130    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.083721 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.083699    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.084228 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.084209    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmg7l\" (UniqueName: \"kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.084455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.084437    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.085014 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.084991    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.085580 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.084161    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.130079 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.124536    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmg7l\" (UniqueName: \"kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.169765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.145822    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.169765 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.146568    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm\") pod \"auth-test-model-4331a7-kserve-d7cb57d5-5lw4g\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.220768 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.220074    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:52.644642 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:52.644607    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"]
Nov 03 23:55:53.652837 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:53.652798    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g" event={"ID":"847cbeb0-5f6a-4666-94ec-a78fca0b0839","Type":"ContainerStarted","Data":"a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137"}
Nov 03 23:55:53.653264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:53.652850    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g" event={"ID":"847cbeb0-5f6a-4666-94ec-a78fca0b0839","Type":"ContainerStarted","Data":"c4baddbc657469f4d79fdb581eabea7b8c50cf0f673fb970fc6d7799f0a4a26e"}
Nov 03 23:55:53.698365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:53.698286    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g" podStartSLOduration=2.698267551 podStartE2EDuration="2.698267551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:53.694461719 +0000 UTC m=+1618.794061865" watchObservedRunningTime="2026-11-03 23:55:53.698267551 +0000 UTC m=+1618.797867696"
Nov 03 23:55:54.220751 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:54.220607    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:55:55.446760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:55.441180    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 03 23:55:56.462118 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:56.462089    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 03 23:55:58.046795 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:55:58.045811    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:55:58.409479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.409394    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"]
Nov 03 23:55:58.423486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.423447    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"]
Nov 03 23:55:58.426495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.423546    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.441554 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.435962    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-2b1f66\""
Nov 03 23:55:58.441807 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.441565    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"]
Nov 03 23:55:58.441807 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.441608    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"]
Nov 03 23:55:58.441807 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.441699    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.448521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.448474    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-2b1f66-dockercfg-snjm4\""
Nov 03 23:55:58.547936 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.547883    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmk6\" (UniqueName: \"kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6\") pod \"payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.548166 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.548073    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.548166 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.548125    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rn9\" (UniqueName: \"kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9\") pod \"payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.548428 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.548398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.652803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.650887    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.652803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.650972    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rn9\" (UniqueName: \"kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9\") pod \"payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.652803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.651253    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.652803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.651415    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmk6\" (UniqueName: \"kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6\") pod \"payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.652803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.651971    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.652803 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.652251    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.671132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.671049    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rn9\" (UniqueName: \"kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9\") pod \"payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:58.671132 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.671054    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmk6\" (UniqueName: \"kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6\") pod \"payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.753760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.750622    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:58.778329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:58.778295    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:59.065800 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:55:59.065773    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5a9dc6_78bc_4333_bf79_119ea7f8c331.slice/crio-7973230355f8a60017aecc768b6511265523d9a682592fbdfe2c16e8c3a84ccd WatchSource:0}: Error finding container 7973230355f8a60017aecc768b6511265523d9a682592fbdfe2c16e8c3a84ccd: Status 404 returned error can't find the container with id 7973230355f8a60017aecc768b6511265523d9a682592fbdfe2c16e8c3a84ccd
Nov 03 23:55:59.066631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.066608    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"]
Nov 03 23:55:59.185060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.176099    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"]
Nov 03 23:55:59.743460 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.743409    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx" event={"ID":"1e5a9dc6-78bc-4333-bf79-119ea7f8c331","Type":"ContainerStarted","Data":"b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab"}
Nov 03 23:55:59.744165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.744140    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:55:59.744319 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.744183    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx" event={"ID":"1e5a9dc6-78bc-4333-bf79-119ea7f8c331","Type":"ContainerStarted","Data":"7973230355f8a60017aecc768b6511265523d9a682592fbdfe2c16e8c3a84ccd"}
Nov 03 23:55:59.745553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.745520    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z" event={"ID":"11504999-0424-42d6-9585-03fb92e13ed2","Type":"ContainerStarted","Data":"2cd21496c9939b4079f9acafb6538be0bf632fefabd2821fa9eb184b1dac9196"}
Nov 03 23:55:59.745656 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.745575    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z" event={"ID":"11504999-0424-42d6-9585-03fb92e13ed2","Type":"ContainerStarted","Data":"fb492a527d9de593a953004c5dc46462aa4ede034194597b6b0351a187bc4cbc"}
Nov 03 23:55:59.745720 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.745687    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:55:59.783415 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.783363    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx" podStartSLOduration=1.783349296 podStartE2EDuration="1.783349296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:59.782256596 +0000 UTC m=+1624.881856742" watchObservedRunningTime="2026-11-03 23:55:59.783349296 +0000 UTC m=+1624.882949439"
Nov 03 23:55:59.821452 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:55:59.821386    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z" podStartSLOduration=1.82136787 podStartE2EDuration="1.82136787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:59.81883772 +0000 UTC m=+1624.918437865" watchObservedRunningTime="2026-11-03 23:55:59.82136787 +0000 UTC m=+1624.920968015"
Nov 03 23:56:02.220785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:02.220747    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:56:02.234730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:02.234704    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:56:02.784618 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:02.784591    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:56:10.759971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.759943    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 03 23:56:10.830772 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.827191    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"]
Nov 03 23:56:10.834563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.834539    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.836339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.836308    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"]
Nov 03 23:56:10.845486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.845465    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-330eb5\"/\"default-dockercfg-dwsxc\""
Nov 03 23:56:10.861855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.861828    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-330eb5\"/\"openshift-service-ca.crt\""
Nov 03 23:56:10.866673 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.866636    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-330eb5\"/\"auth-test-model-330eb5-kserve-self-signed-certs\""
Nov 03 23:56:10.867606 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.867565    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.867716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.867618    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.867716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.867684    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.867858 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.867766    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.867858 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.867819    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkxb\" (UniqueName: \"kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.868089 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.868070    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.962002 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.961965    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-330eb5\"/\"kube-root-ca.crt\""
Nov 03 23:56:10.969886 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.969833    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.969907    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.969940    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.969992    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970221 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.970038    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970393 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.970367    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970495 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.970443    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.970623 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.970514    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkxb\" (UniqueName: \"kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.971038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.971014    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.972426 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.972400    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.973271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.973245    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:10.984836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:10.984812    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkxb\" (UniqueName: \"kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb\") pod \"auth-test-model-330eb5-kserve-798c68db57-kp2d6\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") " pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:11.153369 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:11.153286    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:11.473658 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:56:11.473630    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb75763c_f905_4f8b_9456_6a18f512f2a5.slice/crio-9fa8dcf08662ef5b6f0063eed007ce6acdf988f8a42fce7fc49e3c7c75451d10 WatchSource:0}: Error finding container 9fa8dcf08662ef5b6f0063eed007ce6acdf988f8a42fce7fc49e3c7c75451d10: Status 404 returned error can't find the container with id 9fa8dcf08662ef5b6f0063eed007ce6acdf988f8a42fce7fc49e3c7c75451d10
Nov 03 23:56:11.478553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:11.478483    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"]
Nov 03 23:56:11.769052 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:11.768962    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 03 23:56:11.891166 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:11.891126    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" event={"ID":"fb75763c-f905-4f8b-9456-6a18f512f2a5","Type":"ContainerStarted","Data":"a1284f77b0b351d549fdaf3e253b2d244f6ecca98d6413068aeea655c6ab9f06"}
Nov 03 23:56:11.891166 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:11.891173    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" event={"ID":"fb75763c-f905-4f8b-9456-6a18f512f2a5","Type":"ContainerStarted","Data":"9fa8dcf08662ef5b6f0063eed007ce6acdf988f8a42fce7fc49e3c7c75451d10"}
Nov 03 23:56:11.915589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:11.915537    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" podStartSLOduration=1.915523301 podStartE2EDuration="1.915523301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:11.912075194 +0000 UTC m=+1637.011675339" watchObservedRunningTime="2026-11-03 23:56:11.915523301 +0000 UTC m=+1637.015123442"
Nov 03 23:56:12.154325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:12.154246    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:14.806730 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.806692    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"]
Nov 03 23:56:14.812993 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.812971    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:14.822251 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.821961    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-15b02a\""
Nov 03 23:56:14.825338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.825314    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-15b02a-dockercfg-cbjts\""
Nov 03 23:56:14.831432 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.831408    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"]
Nov 03 23:56:14.837752 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.837711    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:14.837862 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.837770    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hs9c\" (UniqueName: \"kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c\") pod \"payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:14.838302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.838282    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"]
Nov 03 23:56:14.844879 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.844857    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:14.849173 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.849149    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"]
Nov 03 23:56:14.983468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.981648    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:14.983468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.981716    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hs9c\" (UniqueName: \"kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c\") pod \"payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:14.983468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.982003    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj6lw\" (UniqueName: \"kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw\") pod \"payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:14.983468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.982061    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:14.983468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:14.983108    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:15.044761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.039966    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hs9c\" (UniqueName: \"kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c\") pod \"payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:15.083198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.083099    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6lw\" (UniqueName: \"kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw\") pod \"payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:15.083198 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.083170    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:15.084007 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.083982    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:15.102283 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.102249    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj6lw\" (UniqueName: \"kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw\") pod \"payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:15.133226 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.133183    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:15.166113 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.166082    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:15.450271 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:56:15.450238    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea32b661_697a_48fb_8bc2_b950be89f0d4.slice/crio-b05cfb7aa7291ddc57d53c054dabfb2eb11158d546f409c222e9428d789a207e WatchSource:0}: Error finding container b05cfb7aa7291ddc57d53c054dabfb2eb11158d546f409c222e9428d789a207e: Status 404 returned error can't find the container with id b05cfb7aa7291ddc57d53c054dabfb2eb11158d546f409c222e9428d789a207e
Nov 03 23:56:15.452218 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.452199    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"]
Nov 03 23:56:15.520248 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:56:15.520217    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efd4e7c_27b4_4119_b2eb_67d7a512f8ff.slice/crio-a2155ed34972fce624b900e53366691d6560ff461a624f304cac3f5887950da8 WatchSource:0}: Error finding container a2155ed34972fce624b900e53366691d6560ff461a624f304cac3f5887950da8: Status 404 returned error can't find the container with id a2155ed34972fce624b900e53366691d6560ff461a624f304cac3f5887950da8
Nov 03 23:56:15.520690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:15.520668    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"]
Nov 03 23:56:16.053221 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.053180    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw" event={"ID":"ea32b661-697a-48fb-8bc2-b950be89f0d4","Type":"ContainerStarted","Data":"c9e36cd2d992e9ef97c149317f283fed43e3edf3865d55f90e2cc6654979b43c"}
Nov 03 23:56:16.053893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.053864    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw" event={"ID":"ea32b661-697a-48fb-8bc2-b950be89f0d4","Type":"ContainerStarted","Data":"b05cfb7aa7291ddc57d53c054dabfb2eb11158d546f409c222e9428d789a207e"}
Nov 03 23:56:16.054213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.054196    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:16.057255 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.057221    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r" event={"ID":"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff","Type":"ContainerStarted","Data":"9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775"}
Nov 03 23:56:16.057366 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.057270    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r" event={"ID":"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff","Type":"ContainerStarted","Data":"a2155ed34972fce624b900e53366691d6560ff461a624f304cac3f5887950da8"}
Nov 03 23:56:16.057366 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.057310    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:16.083271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.083218    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw" podStartSLOduration=2.083203346 podStartE2EDuration="2.083203346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:16.077891941 +0000 UTC m=+1641.177492097" watchObservedRunningTime="2026-11-03 23:56:16.083203346 +0000 UTC m=+1641.182803489"
Nov 03 23:56:16.110434 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:16.110371    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r" podStartSLOduration=2.11035351 podStartE2EDuration="2.11035351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:16.107522841 +0000 UTC m=+1641.207122986" watchObservedRunningTime="2026-11-03 23:56:16.11035351 +0000 UTC m=+1641.209953654"
Nov 03 23:56:19.764773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.763522    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"]
Nov 03 23:56:19.772053 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.772025    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.793304 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.793273    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"]
Nov 03 23:56:19.824762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.793894    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"kube-root-ca.crt\""
Nov 03 23:56:19.824762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.793962    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"auth-test-model-398158-kserve-self-signed-certs\""
Nov 03 23:56:19.824762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.794008    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"openshift-service-ca.crt\""
Nov 03 23:56:19.824762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.796151    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"default-dockercfg-gwkft\""
Nov 03 23:56:19.870056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.870024    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.870237 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.870214    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7t6\" (UniqueName: \"kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.870350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.870333    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.870468 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.870450    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.870604 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.870581    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.870664 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.870632    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972056 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972014    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972106    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972141    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972210    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972319    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7t6\" (UniqueName: \"kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972390 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972382    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972456    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972470    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.972777 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.972728    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.975082 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.975049    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.975303 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.975286    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:19.986431 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:19.986400    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7t6\" (UniqueName: \"kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6\") pod \"auth-test-model-398158-kserve-867bd6577c-xs5gw\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") " pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:20.136763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:20.136360    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:20.580502 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:56:20.580470    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb013efee_c523_4cc4_bd29_6e295477f328.slice/crio-6bf7dd9c7531462cfd3557d457334febee04c256a7d0d3c0184a96eaf4fccd9b WatchSource:0}: Error finding container 6bf7dd9c7531462cfd3557d457334febee04c256a7d0d3c0184a96eaf4fccd9b: Status 404 returned error can't find the container with id 6bf7dd9c7531462cfd3557d457334febee04c256a7d0d3c0184a96eaf4fccd9b
Nov 03 23:56:20.581116 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:20.581023    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"]
Nov 03 23:56:21.154419 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:21.154383    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:21.167807 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:21.167779    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:21.195892 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:21.195849    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" event={"ID":"b013efee-c523-4cc4-bd29-6e295477f328","Type":"ContainerStarted","Data":"aa40564615c74146f3bbd56d54eb330746f3051a1ba818c1009f1c37fc411a7e"}
Nov 03 23:56:21.196149 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:21.196127    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" event={"ID":"b013efee-c523-4cc4-bd29-6e295477f328","Type":"ContainerStarted","Data":"6bf7dd9c7531462cfd3557d457334febee04c256a7d0d3c0184a96eaf4fccd9b"}
Nov 03 23:56:21.219212 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:21.219165    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 03 23:56:21.220155 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:21.220100    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" podStartSLOduration=2.22008251 podStartE2EDuration="2.22008251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:21.218195606 +0000 UTC m=+1646.317795752" watchObservedRunningTime="2026-11-03 23:56:21.22008251 +0000 UTC m=+1646.319682655"
Nov 03 23:56:22.140875 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:22.140827    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:27.069194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:27.069160    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 03 23:56:27.069691 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:27.069231    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 03 23:56:28.026957 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:56:28.026923    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:56:29.048563 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.048531    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"]
Nov 03 23:56:29.078581 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.078537    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.083536 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.083492    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"]
Nov 03 23:56:29.113668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.113637    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"kube-root-ca.crt\""
Nov 03 23:56:29.114113 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.114084    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"default-dockercfg-9h4s2\""
Nov 03 23:56:29.114985 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.114964    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"openshift-service-ca.crt\""
Nov 03 23:56:29.118313 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.118288    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"auth-test-model-58b75d-kserve-self-signed-certs\""
Nov 03 23:56:29.226049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.225890    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.226049 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.226020    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.226440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.226167    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4pnk\" (UniqueName: \"kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.226440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.226281    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.226440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.226359    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.226440 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.226422    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.336921    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.337032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.337124    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4pnk\" (UniqueName: \"kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.337229    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.337306    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.337383    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.337860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.337491    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.338406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.338291    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.341761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.338549    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.341761 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.340623    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.351848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.348575    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.362760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.356452    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4pnk\" (UniqueName: \"kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk\") pod \"auth-test-model-58b75d-kserve-58556bb75c-hs7px\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.409763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.409104    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:29.779760 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:29.778984    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"]
Nov 03 23:56:30.152377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.143642    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:30.167121 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.167090    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:30.327932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.327891    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px" event={"ID":"4f0cca6f-ea89-4754-8eed-3d8069713fd1","Type":"ContainerStarted","Data":"00899d3a26c84ddd0d415a0b4bd71b8ef407fad39aa2bb951f289224141cab64"}
Nov 03 23:56:30.327932 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.327948    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px" event={"ID":"4f0cca6f-ea89-4754-8eed-3d8069713fd1","Type":"ContainerStarted","Data":"1cc3ed2c5c4fdbb93b7e6a8f4fa705a28ba3b28099705a703b79d03db1c52e8f"}
Nov 03 23:56:30.342815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.340966    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:56:30.372114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.372052    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px" podStartSLOduration=1.372034655 podStartE2EDuration="1.372034655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:30.349163393 +0000 UTC m=+1655.448763537" watchObservedRunningTime="2026-11-03 23:56:30.372034655 +0000 UTC m=+1655.471634800"
Nov 03 23:56:30.410480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:30.410333    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:39.409977 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:39.409886    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:39.426539 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:39.426502    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:39.446346 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:39.446315    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 03 23:56:47.031362 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:47.031328    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"]
Nov 03 23:56:47.039040 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:47.039012    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d50c6f-hgbfl"]
Nov 03 23:56:49.021855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:49.021814    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d1cd17-a7ef-4111-92b3-9344f9d4c26b" path="/var/lib/kubelet/pods/a3d1cd17-a7ef-4111-92b3-9344f9d4c26b/volumes"
Nov 03 23:56:51.989643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.989594    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"]
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.996840    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.997446    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"]
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.999186    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnw5\" (UniqueName: \"kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.999221    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.999254    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.999297    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.999418    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.000906 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:51.999498    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.010129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.010107    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"kube-root-ca.crt\""
Nov 03 23:56:52.012931 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.012910    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"default-dockercfg-xb2f8\""
Nov 03 23:56:52.013127 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.013113    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"openshift-service-ca.crt\""
Nov 03 23:56:52.013355 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.013337    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"auth-test-model-b3bea7-kserve-self-signed-certs\""
Nov 03 23:56:52.101093 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101050    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.101302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101202    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnw5\" (UniqueName: \"kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.101302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101235    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.101302 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101265    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.101474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101307    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.101474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101408    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.101975 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.101943    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.102114 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.102019    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.102174 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.102159    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.104168 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.104139    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.104990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.104957    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.121716 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.121687    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnw5\" (UniqueName: \"kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5\") pod \"auth-test-model-b3bea7-kserve-79fbb85c89-266t9\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.333112 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.332788    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:52.686411 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:56:52.686379    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee25e7c_e20b_406d_9b79_8f552b0e639e.slice/crio-1c356732f3c184e11e2194eded9e5b0a1ff11a4a74e3468c909bbd9040437b15 WatchSource:0}: Error finding container 1c356732f3c184e11e2194eded9e5b0a1ff11a4a74e3468c909bbd9040437b15: Status 404 returned error can't find the container with id 1c356732f3c184e11e2194eded9e5b0a1ff11a4a74e3468c909bbd9040437b15
Nov 03 23:56:52.687570 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:52.687543    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"]
Nov 03 23:56:53.574045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:53.574006    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9" event={"ID":"eee25e7c-e20b-406d-9b79-8f552b0e639e","Type":"ContainerStarted","Data":"ec1c4bc71d4c716c2c5b3c31fbe83c6f4c057c7ee8ce7e97e14b45ddded0869b"}
Nov 03 23:56:53.574530 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:53.574061    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9" event={"ID":"eee25e7c-e20b-406d-9b79-8f552b0e639e","Type":"ContainerStarted","Data":"1c356732f3c184e11e2194eded9e5b0a1ff11a4a74e3468c909bbd9040437b15"}
Nov 03 23:56:53.595038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:53.594990    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9" podStartSLOduration=2.594973054 podStartE2EDuration="2.594973054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:53.594366713 +0000 UTC m=+1678.693966857" watchObservedRunningTime="2026-11-03 23:56:53.594973054 +0000 UTC m=+1678.694573197"
Nov 03 23:56:54.334224 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:56:54.334187    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:56:58.029112 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:56:58.029069    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:57:01.729834 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.729794    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"]
Nov 03 23:57:01.757076 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.757047    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"]
Nov 03 23:57:01.757260 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.757141    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.770646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.770330    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"default-dockercfg-qtzvl\""
Nov 03 23:57:01.770646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.770395    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"openshift-service-ca.crt\""
Nov 03 23:57:01.770646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.770453    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"kube-root-ca.crt\""
Nov 03 23:57:01.770646 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.770600    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"auth-test-model-8093f5-kserve-self-signed-certs\""
Nov 03 23:57:01.871534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.871498    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.871700 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.871634    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xcl\" (UniqueName: \"kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.871700 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.871688    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.871860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.871848    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.871961 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.871948    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.871999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.871987    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.976211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.975664    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.976211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.975805    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.976211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.975898    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xcl\" (UniqueName: \"kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.976211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.975960    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.976211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.976046    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.976211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.976173    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.984844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.977472    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.984844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.978048    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:01.984844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:01.978290    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:02.008085 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.008050    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:02.016631 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.016587    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xcl\" (UniqueName: \"kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:02.036773 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.035321    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs\") pod \"auth-test-model-8093f5-kserve-589f895997-hszmc\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:02.089668 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.089564    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:02.334494 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.334400    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:57:02.352175 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.352136    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:57:02.455307 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.455275    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"]
Nov 03 23:57:02.664097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.663998    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" event={"ID":"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328","Type":"ContainerStarted","Data":"e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da"}
Nov 03 23:57:02.664097 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.664061    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" event={"ID":"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328","Type":"ContainerStarted","Data":"71d2331d0750a6b0e0727dae932c4e79d5c557899907c7ec051459aaa7b8f973"}
Nov 03 23:57:02.682032 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.682002    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 03 23:57:02.728916 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:02.728857    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" podStartSLOduration=1.7288387219999999 podStartE2EDuration="1.728838722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:57:02.724707646 +0000 UTC m=+1687.824307806" watchObservedRunningTime="2026-11-03 23:57:02.728838722 +0000 UTC m=+1687.828438872"
Nov 03 23:57:03.089789 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:03.089720    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:12.090595 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:12.090552    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:12.110567 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:12.110536    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:12.788395 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:12.788362    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 03 23:57:19.033131 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:19.033086    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"]
Nov 03 23:57:19.038191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:19.038166    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d513be-6pqpf"]
Nov 03 23:57:21.023147 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:21.023102    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe83b3f-4a56-44e9-a168-51a24f031d52" path="/var/lib/kubelet/pods/ffe83b3f-4a56-44e9-a168-51a24f031d52/volumes"
Nov 03 23:57:28.025632 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:57:28.025589    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:57:36.033455 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:36.033420    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"]
Nov 03 23:57:36.038666 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:36.038636    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582ae5d1-lfv5v"]
Nov 03 23:57:37.019636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:37.019598    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8baf683e-b684-479c-9a75-7e405002c4a2" path="/var/lib/kubelet/pods/8baf683e-b684-479c-9a75-7e405002c4a2/volumes"
Nov 03 23:57:48.034698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:48.034657    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"]
Nov 03 23:57:48.039303 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:48.039271    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89227a-rxvkg"]
Nov 03 23:57:49.019420 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:49.019383    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0a0f5d-ef70-45cf-bca9-92f09faeee6c" path="/var/lib/kubelet/pods/4d0a0f5d-ef70-45cf-bca9-92f09faeee6c/volumes"
Nov 03 23:57:50.034577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:50.034541    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"]
Nov 03 23:57:50.041929 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:50.041891    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478eb918-t87r7"]
Nov 03 23:57:51.025257 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:51.025213    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6e0576-b847-49c6-b9e6-0de41645887a" path="/var/lib/kubelet/pods/fb6e0576-b847-49c6-b9e6-0de41645887a/volumes"
Nov 03 23:57:51.035271 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:51.035240    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"]
Nov 03 23:57:51.042451 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:51.042422    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ae5419-cp7fx"]
Nov 03 23:57:53.019446 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:53.019406    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389c764a-caab-4391-96b7-91fa6d2362cd" path="/var/lib/kubelet/pods/389c764a-caab-4391-96b7-91fa6d2362cd/volumes"
Nov 03 23:57:56.479570 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:57:56.479537    5074 scope.go:122] "RemoveContainer" containerID="f8af3ec3d22756fcbcab64f17e70145e2d48581246774fed013956e0697584a3"
Nov 03 23:57:58.023487 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:57:58.023445    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:58:01.031242 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:01.031205    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"]
Nov 03 23:58:01.038657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:01.038626    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68b66921-zlcws"]
Nov 03 23:58:03.021078 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:03.021037    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe453ec-b910-4233-a3ed-43f3507cf73a" path="/var/lib/kubelet/pods/7fe453ec-b910-4233-a3ed-43f3507cf73a/volumes"
Nov 03 23:58:04.033156 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:04.033124    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"]
Nov 03 23:58:04.040230 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:04.040202    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b41dec-tzkrb"]
Nov 03 23:58:04.047023 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:04.046998    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"]
Nov 03 23:58:04.052749 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:04.052710    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb060ae-bqkbg"]
Nov 03 23:58:05.032467 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:05.032433    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813afa38-726c-48a5-9b99-c47ac49db7ad" path="/var/lib/kubelet/pods/813afa38-726c-48a5-9b99-c47ac49db7ad/volumes"
Nov 03 23:58:05.033164 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:05.033146    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa51b005-48b9-49d9-91a2-182dff56564a" path="/var/lib/kubelet/pods/fa51b005-48b9-49d9-91a2-182dff56564a/volumes"
Nov 03 23:58:24.032546 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:24.032507    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"]
Nov 03 23:58:24.037001 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:24.036970    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1018b3a0-46wvl"]
Nov 03 23:58:25.024326 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:25.024279    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e008ab4f-315c-49f6-bd78-a457887d86bb" path="/var/lib/kubelet/pods/e008ab4f-315c-49f6-bd78-a457887d86bb/volumes"
Nov 03 23:58:28.031802 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:28.031406    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"]
Nov 03 23:58:28.032263 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:58:28.031928    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:58:28.036263 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:28.036235    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cf60-tg68g"]
Nov 03 23:58:28.043404 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:28.043380    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"]
Nov 03 23:58:28.049282 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:28.049260    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cac345a-krk89"]
Nov 03 23:58:29.020317 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:29.020283    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641a5425-f53b-4991-b168-d37ece30dad7" path="/var/lib/kubelet/pods/641a5425-f53b-4991-b168-d37ece30dad7/volumes"
Nov 03 23:58:29.020855 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:29.020835    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacd51a6-58d6-433a-945d-96408d93fcbe" path="/var/lib/kubelet/pods/bacd51a6-58d6-433a-945d-96408d93fcbe/volumes"
Nov 03 23:58:34.029415 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:34.029368    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"]
Nov 03 23:58:34.033867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:34.033840    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-072673a8-svpns"]
Nov 03 23:58:35.023088 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:35.023052    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf6c359-3890-493e-936d-35c4f79b7bba" path="/var/lib/kubelet/pods/bdf6c359-3890-493e-936d-35c4f79b7bba/volumes"
Nov 03 23:58:41.029401 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:41.029359    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"]
Nov 03 23:58:41.036763 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:41.035727    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6757a157-bgp64"]
Nov 03 23:58:43.021988 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:43.021947    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783ff9cd-6a8a-4e96-9d1a-5ed038bdb892" path="/var/lib/kubelet/pods/783ff9cd-6a8a-4e96-9d1a-5ed038bdb892/volumes"
Nov 03 23:58:51.043586 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:51.043540    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"]
Nov 03 23:58:51.048816 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:51.048782    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4e3d9c-s6dhv"]
Nov 03 23:58:53.020672 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:53.020633    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdcda55-6aef-4977-9872-fbef4244524c" path="/var/lib/kubelet/pods/bbdcda55-6aef-4977-9872-fbef4244524c/volumes"
Nov 03 23:58:55.491832 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:55.491802    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:58:55.543029 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:55.543003    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 03 23:58:56.514184 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:56.514141    5074 scope.go:122] "RemoveContainer" containerID="5087397d56ca7a4eeba18f26113c9b74f77384c0ff2ffd85fc7d9b0bed3c891a"
Nov 03 23:58:56.542626 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:56.542592    5074 scope.go:122] "RemoveContainer" containerID="e330643b5ca53e30a67219f9114b7844974a05ae0e8628e160482e8a4152ec20"
Nov 03 23:58:56.556844 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:56.556808    5074 scope.go:122] "RemoveContainer" containerID="4b48a20adc7f251677e48f1984b9915dfc768e7e36ad9eea3539237244d65889"
Nov 03 23:58:56.571148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:56.571106    5074 scope.go:122] "RemoveContainer" containerID="d1fafe77560d380c0f7ec9eec82bd82032145b9c3bc16fc4f9f8069163865d3d"
Nov 03 23:58:56.584524 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:56.584500    5074 scope.go:122] "RemoveContainer" containerID="cee23b77916d7536b011bca3df2ea62018dba1ffef834993d1e38549648d6941"
Nov 03 23:58:58.025983 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:58:58.025942    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:58:58.035553 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:58.035526    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"]
Nov 03 23:58:58.041228 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:58.041205    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701cb87e-nf4rm"]
Nov 03 23:58:59.019684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:58:59.019651    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b81f83-65db-4c81-a068-a35310a0f9dc" path="/var/lib/kubelet/pods/98b81f83-65db-4c81-a068-a35310a0f9dc/volumes"
Nov 03 23:59:01.441725 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.441693    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"]
Nov 03 23:59:01.448384 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.448351    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.454783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.454722    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:59:01.455010 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.454986    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:59:01.455548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.455522    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"]
Nov 03 23:59:01.522778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.522695    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.523045 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.523024    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.523210 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.523191    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhkn\" (UniqueName: \"kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.625310 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.625276    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.625521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.625400    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhkn\" (UniqueName: \"kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.625521 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.625514    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.626424 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.626397    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.628139 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.628103    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.639829 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.639779    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhkn\" (UniqueName: \"kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn\") pod \"maas-api-revoke-sub-ef2c0c81-727hf\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:01.767270 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:01.767187    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:02.069410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:02.069379    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"]
Nov 03 23:59:02.071379 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:02.071347    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd71f8d_6d18_4d1b_a0e6_7c6751057cac.slice/crio-f7a55162b32d3457a08aa96b83ae034ad3888be6509cf8558c9bbf2290e9e7c8 WatchSource:0}: Error finding container f7a55162b32d3457a08aa96b83ae034ad3888be6509cf8558c9bbf2290e9e7c8: Status 404 returned error can't find the container with id f7a55162b32d3457a08aa96b83ae034ad3888be6509cf8558c9bbf2290e9e7c8
Nov 03 23:59:02.866311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:02.866270    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf" event={"ID":"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac","Type":"ContainerStarted","Data":"298f6670f44b14d9255aebd9736e4e099e378ea041592e8b702a5618f8423d9f"}
Nov 03 23:59:02.866755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:02.866337    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf" event={"ID":"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac","Type":"ContainerStarted","Data":"f7a55162b32d3457a08aa96b83ae034ad3888be6509cf8558c9bbf2290e9e7c8"}
Nov 03 23:59:02.889515 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:02.889461    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf" podStartSLOduration=1.88944859 podStartE2EDuration="1.88944859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:02.885646279 +0000 UTC m=+1807.985246424" watchObservedRunningTime="2026-11-03 23:59:02.88944859 +0000 UTC m=+1807.989048748"
Nov 03 23:59:03.032401 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:03.032365    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"]
Nov 03 23:59:03.039288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:03.039258    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"]
Nov 03 23:59:03.045796 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:03.045733    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c65eb1-8sbm5"]
Nov 03 23:59:03.051543 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:03.051517    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a59128-zjqjv"]
Nov 03 23:59:03.876755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:03.876711    5074 generic.go:356] "Generic (PLEG): container finished" podID="fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" containerID="298f6670f44b14d9255aebd9736e4e099e378ea041592e8b702a5618f8423d9f" exitCode=0
Nov 03 23:59:03.877246 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:03.876798    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf" event={"ID":"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac","Type":"ContainerDied","Data":"298f6670f44b14d9255aebd9736e4e099e378ea041592e8b702a5618f8423d9f"}
Nov 03 23:59:05.033793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.032095    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0" path="/var/lib/kubelet/pods/178d8d46-053a-4ca3-8bb0-01c6d0a7c8c0/volumes"
Nov 03 23:59:05.033793 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.032768    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5aa0b5b-f72f-4d96-a2c6-1be9dae50553" path="/var/lib/kubelet/pods/d5aa0b5b-f72f-4d96-a2c6-1be9dae50553/volumes"
Nov 03 23:59:05.106190 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.106164    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:05.185397 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.185362    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn\" (UniqueName: \"kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn\") pod \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") "
Nov 03 23:59:05.185589 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.185544    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca\") pod \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") "
Nov 03 23:59:05.185674 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.185658    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca\") pod \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\" (UID: \"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac\") "
Nov 03 23:59:05.186048 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.186013    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca" pod "fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" (UID: "fcd71f8d-6d18-4d1b-a0e6-7c6751057cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:59:05.186329 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.186308    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:05.188301 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.188272    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn" pod "fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" (UID: "fcd71f8d-6d18-4d1b-a0e6-7c6751057cac"). InnerVolumeSpecName "kube-api-access-8nhkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:05.188477 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.188455    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca" pod "fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" (UID: "fcd71f8d-6d18-4d1b-a0e6-7c6751057cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:59:05.288046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.288014    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nhkn\" (UniqueName: \"kubernetes.io/projected/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-kube-api-access-8nhkn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:05.288046 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.288043    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:05.921609 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.921575    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"
Nov 03 23:59:05.921815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.921637    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf" event={"ID":"fcd71f8d-6d18-4d1b-a0e6-7c6751057cac","Type":"ContainerDied","Data":"f7a55162b32d3457a08aa96b83ae034ad3888be6509cf8558c9bbf2290e9e7c8"}
Nov 03 23:59:05.921815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:05.921711    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a55162b32d3457a08aa96b83ae034ad3888be6509cf8558c9bbf2290e9e7c8"
Nov 03 23:59:11.747474 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.747391    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"]
Nov 03 23:59:11.748189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.748172    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" containerName="revoke-keys"
Nov 03 23:59:11.748243 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.748191    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" containerName="revoke-keys"
Nov 03 23:59:11.748336 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.748327    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" containerName="revoke-keys"
Nov 03 23:59:11.752335 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.752310    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.758657 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.758630    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"]
Nov 03 23:59:11.759100 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.759080    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:59:11.759178 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.759100    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:59:11.896208 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.896165    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.896405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.896367    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.896405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.896387    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zb4\" (UniqueName: \"kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.998130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.998045    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.998130 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.998084    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zb4\" (UniqueName: \"kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.998365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.998305    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:11.998854 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:11.998831    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:12.000924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:12.000896    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:12.011611 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:12.011585    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zb4\" (UniqueName: \"kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4\") pod \"maas-api-revoke-sub-c01cce37-grlhg\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:12.070144 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:12.070109    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:12.376583 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:12.376553    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"]
Nov 03 23:59:12.376702 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:12.376660    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc85bf65_dbd9_428e_854d_4984fa6c0926.slice/crio-16338cc293e94edafb1226b964976db672c89b71f647defe0e040dfd1ace1d94 WatchSource:0}: Error finding container 16338cc293e94edafb1226b964976db672c89b71f647defe0e040dfd1ace1d94: Status 404 returned error can't find the container with id 16338cc293e94edafb1226b964976db672c89b71f647defe0e040dfd1ace1d94
Nov 03 23:59:12.986924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:12.986822    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg" event={"ID":"dc85bf65-dbd9-428e-854d-4984fa6c0926","Type":"ContainerStarted","Data":"916914e51b8d95c01ebf22b0de0ba28aa1b3946f61e4ae39d19294f4366b45f2"}
Nov 03 23:59:12.986924 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:12.986897    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg" event={"ID":"dc85bf65-dbd9-428e-854d-4984fa6c0926","Type":"ContainerStarted","Data":"16338cc293e94edafb1226b964976db672c89b71f647defe0e040dfd1ace1d94"}
Nov 03 23:59:13.996964 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:13.996925    5074 generic.go:356] "Generic (PLEG): container finished" podID="dc85bf65-dbd9-428e-854d-4984fa6c0926" containerID="916914e51b8d95c01ebf22b0de0ba28aa1b3946f61e4ae39d19294f4366b45f2" exitCode=0
Nov 03 23:59:13.997406 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:13.997007    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg" event={"ID":"dc85bf65-dbd9-428e-854d-4984fa6c0926","Type":"ContainerDied","Data":"916914e51b8d95c01ebf22b0de0ba28aa1b3946f61e4ae39d19294f4366b45f2"}
Nov 03 23:59:15.236377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.236346    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:15.249333 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.249300    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca\") pod \"dc85bf65-dbd9-428e-854d-4984fa6c0926\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") "
Nov 03 23:59:15.249559 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.249543    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca\") pod \"dc85bf65-dbd9-428e-854d-4984fa6c0926\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") "
Nov 03 23:59:15.249659 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.249647    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4\" (UniqueName: \"kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4\") pod \"dc85bf65-dbd9-428e-854d-4984fa6c0926\" (UID: \"dc85bf65-dbd9-428e-854d-4984fa6c0926\") "
Nov 03 23:59:15.250875 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.250366    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca" pod "dc85bf65-dbd9-428e-854d-4984fa6c0926" (UID: "dc85bf65-dbd9-428e-854d-4984fa6c0926"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:59:15.252867 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.252841    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca" pod "dc85bf65-dbd9-428e-854d-4984fa6c0926" (UID: "dc85bf65-dbd9-428e-854d-4984fa6c0926"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:59:15.253225 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.253204    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4" pod "dc85bf65-dbd9-428e-854d-4984fa6c0926" (UID: "dc85bf65-dbd9-428e-854d-4984fa6c0926"). InnerVolumeSpecName "kube-api-access-v8zb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:15.352213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.352171    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:15.352213 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.352208    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bf65-dbd9-428e-854d-4984fa6c0926-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:15.352486 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:15.352223    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8zb4\" (UniqueName: \"kubernetes.io/projected/dc85bf65-dbd9-428e-854d-4984fa6c0926-kube-api-access-v8zb4\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:16.016732 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:16.016702    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"
Nov 03 23:59:16.016732 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:16.016719    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg" event={"ID":"dc85bf65-dbd9-428e-854d-4984fa6c0926","Type":"ContainerDied","Data":"16338cc293e94edafb1226b964976db672c89b71f647defe0e040dfd1ace1d94"}
Nov 03 23:59:16.016997 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:16.016782    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16338cc293e94edafb1226b964976db672c89b71f647defe0e040dfd1ace1d94"
Nov 03 23:59:20.029291 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:20.029253    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"]
Nov 03 23:59:20.033516 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:20.033484    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a09800-d8fpb"]
Nov 03 23:59:21.021321 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:21.021284    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d9c722-06ab-4521-9f5e-c8716dd96e69" path="/var/lib/kubelet/pods/86d9c722-06ab-4521-9f5e-c8716dd96e69/volumes"
Nov 03 23:59:21.028192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:21.028158    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"]
Nov 03 23:59:21.032057 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:21.032030    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ef888-sds9s"]
Nov 03 23:59:23.021881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:23.021849    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b115faa0-6316-4e24-9a24-d693e5711d1a" path="/var/lib/kubelet/pods/b115faa0-6316-4e24-9a24-d693e5711d1a/volumes"
Nov 03 23:59:23.850587 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:23.850548    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"]
Nov 03 23:59:28.027097 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:28.027059    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:59:29.141351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.141312    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"]
Nov 03 23:59:29.392876 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.392783    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7311-bb5953"]
Nov 03 23:59:29.394307 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.394275    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dc85bf65-dbd9-428e-854d-4984fa6c0926" containerName="revoke-keys"
Nov 03 23:59:29.394484 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.394469    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc85bf65-dbd9-428e-854d-4984fa6c0926" containerName="revoke-keys"
Nov 03 23:59:29.394838 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.394821    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc85bf65-dbd9-428e-854d-4984fa6c0926" containerName="revoke-keys"
Nov 03 23:59:29.414955 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.414922    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-bb5953"]
Nov 03 23:59:29.415141 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.415049    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:29.422817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.422327    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hxqlx\""
Nov 03 23:59:29.489548 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.489508    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kxm\" (UniqueName: \"kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm\") pod \"test-curl-7311-bb5953\" (UID: \"a758da25-0e84-46f4-8664-28e4d347c77a\") " pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:29.591203 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.591169    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kxm\" (UniqueName: \"kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm\") pod \"test-curl-7311-bb5953\" (UID: \"a758da25-0e84-46f4-8664-28e4d347c77a\") " pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:29.607143 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.607114    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kxm\" (UniqueName: \"kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm\") pod \"test-curl-7311-bb5953\" (UID: \"a758da25-0e84-46f4-8664-28e4d347c77a\") " pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:29.772211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:29.772116    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:30.066645 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:30.066441    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-bb5953"]
Nov 03 23:59:30.067963 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:30.067888    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda758da25_0e84_46f4_8664_28e4d347c77a.slice/crio-46248c612bc6c4b8552fc16865a70b4c24f2783fb2115c5143bcf7ff7ea047a2 WatchSource:0}: Error finding container 46248c612bc6c4b8552fc16865a70b4c24f2783fb2115c5143bcf7ff7ea047a2: Status 404 returned error can't find the container with id 46248c612bc6c4b8552fc16865a70b4c24f2783fb2115c5143bcf7ff7ea047a2
Nov 03 23:59:30.160211 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:30.160181    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-bb5953" event={"ID":"a758da25-0e84-46f4-8664-28e4d347c77a","Type":"ContainerStarted","Data":"36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634"}
Nov 03 23:59:30.160534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:30.160222    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-bb5953" event={"ID":"a758da25-0e84-46f4-8664-28e4d347c77a","Type":"ContainerStarted","Data":"46248c612bc6c4b8552fc16865a70b4c24f2783fb2115c5143bcf7ff7ea047a2"}
Nov 03 23:59:30.178348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:30.178304    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7311-bb5953" podStartSLOduration=1.178291717 podStartE2EDuration="1.178291717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:30.177374775 +0000 UTC m=+1835.276974918" watchObservedRunningTime="2026-11-03 23:59:30.178291717 +0000 UTC m=+1835.277891861"
Nov 03 23:59:30.937684 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:30.936817    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7311-bb5953"]
Nov 03 23:59:30.944822 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:30.944245    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7311-bb5953"]
Nov 03 23:59:31.175410 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.175369    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7311-cf2d4a"]
Nov 03 23:59:31.176385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.176358    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a758da25-0e84-46f4-8664-28e4d347c77a" containerName="test-curl-7311-bb5953"
Nov 03 23:59:31.176385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.176384    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a758da25-0e84-46f4-8664-28e4d347c77a" containerName="test-curl-7311-bb5953"
Nov 03 23:59:31.176608 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.176592    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a758da25-0e84-46f4-8664-28e4d347c77a" containerName="test-curl-7311-bb5953"
Nov 03 23:59:31.180523 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.180499    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:31.187989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.187908    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-cf2d4a"]
Nov 03 23:59:31.344726 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.337858    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7lp\" (UniqueName: \"kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp\") pod \"test-curl-7311-cf2d4a\" (UID: \"795dbf46-d56e-4bdc-9183-2114f429d2ce\") " pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:31.440194 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.440099    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7lp\" (UniqueName: \"kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp\") pod \"test-curl-7311-cf2d4a\" (UID: \"795dbf46-d56e-4bdc-9183-2114f429d2ce\") " pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:31.453348 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.453316    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7lp\" (UniqueName: \"kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp\") pod \"test-curl-7311-cf2d4a\" (UID: \"795dbf46-d56e-4bdc-9183-2114f429d2ce\") " pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:31.499864 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.499836    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:31.784201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:31.784169    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-cf2d4a"]
Nov 03 23:59:31.786623 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:31.786597    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod795dbf46_d56e_4bdc_9183_2114f429d2ce.slice/crio-d5fbb6cec2f15cd4a28189e867f107e16211b29aa08661ace9858fca33eb8c92 WatchSource:0}: Error finding container d5fbb6cec2f15cd4a28189e867f107e16211b29aa08661ace9858fca33eb8c92: Status 404 returned error can't find the container with id d5fbb6cec2f15cd4a28189e867f107e16211b29aa08661ace9858fca33eb8c92
Nov 03 23:59:32.031038 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.030932    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"]
Nov 03 23:59:32.036957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.036920    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e27c18-6mt4g"]
Nov 03 23:59:32.185437 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.185389    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-cf2d4a" event={"ID":"795dbf46-d56e-4bdc-9183-2114f429d2ce","Type":"ContainerStarted","Data":"ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859"}
Nov 03 23:59:32.185953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.185446    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-cf2d4a" event={"ID":"795dbf46-d56e-4bdc-9183-2114f429d2ce","Type":"ContainerStarted","Data":"d5fbb6cec2f15cd4a28189e867f107e16211b29aa08661ace9858fca33eb8c92"}
Nov 03 23:59:32.185953 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.185441    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7311-bb5953" podUID="a758da25-0e84-46f4-8664-28e4d347c77a" containerName="test-curl-7311-bb5953" containerID="cri-o://36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634" gracePeriod=2
Nov 03 23:59:32.204192 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.204126    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7311-cf2d4a" podStartSLOduration=1.204113408 podStartE2EDuration="1.204113408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:32.202016549 +0000 UTC m=+1837.301616694" watchObservedRunningTime="2026-11-03 23:59:32.204113408 +0000 UTC m=+1837.303713550"
Nov 03 23:59:32.952060 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.952012    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7311-cf2d4a"]
Nov 03 23:59:32.960759 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:32.957695    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7311-cf2d4a"]
Nov 03 23:59:33.023643 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.023598    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55541821-33a8-438f-9e36-85b78645aeaf" path="/var/lib/kubelet/pods/55541821-33a8-438f-9e36-85b78645aeaf/volumes"
Nov 03 23:59:33.107836 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.107797    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7311-b07ced"]
Nov 03 23:59:33.108956 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.108935    5074 cpu_manager.go:414] "containerMap: removing container" podUID="795dbf46-d56e-4bdc-9183-2114f429d2ce" containerName="test-curl-7311-cf2d4a"
Nov 03 23:59:33.109027 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.108958    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795dbf46-d56e-4bdc-9183-2114f429d2ce" containerName="test-curl-7311-cf2d4a"
Nov 03 23:59:33.109162 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.109150    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="795dbf46-d56e-4bdc-9183-2114f429d2ce" containerName="test-curl-7311-cf2d4a"
Nov 03 23:59:33.114957 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.114928    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:33.118311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.118286    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-b07ced"]
Nov 03 23:59:33.184615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.184576    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgvr\" (UniqueName: \"kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr\") pod \"test-curl-7311-b07ced\" (UID: \"900e8995-64b9-4c37-81cd-c0bcd0b4369f\") " pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:33.286925 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.286838    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgvr\" (UniqueName: \"kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr\") pod \"test-curl-7311-b07ced\" (UID: \"900e8995-64b9-4c37-81cd-c0bcd0b4369f\") " pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:33.301651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.301621    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgvr\" (UniqueName: \"kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr\") pod \"test-curl-7311-b07ced\" (UID: \"900e8995-64b9-4c37-81cd-c0bcd0b4369f\") " pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:33.456015 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.455978    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:33.808143 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:33.808114    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-b07ced"]
Nov 03 23:59:34.204774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.204699    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-b07ced" event={"ID":"900e8995-64b9-4c37-81cd-c0bcd0b4369f","Type":"ContainerStarted","Data":"e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f"}
Nov 03 23:59:34.205377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.204784    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-b07ced" event={"ID":"900e8995-64b9-4c37-81cd-c0bcd0b4369f","Type":"ContainerStarted","Data":"70fa17ddf080daaaa5b3b75bb5dcd0f8b8590b8a2ab6803c68c051ca07972df5"}
Nov 03 23:59:34.205377 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.204956    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7311-cf2d4a" podUID="795dbf46-d56e-4bdc-9183-2114f429d2ce" containerName="test-curl-7311-cf2d4a" containerID="cri-o://ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859" gracePeriod=2
Nov 03 23:59:34.228127 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.228080    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7311-b07ced" podStartSLOduration=1.22806714 podStartE2EDuration="1.22806714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:34.22282553 +0000 UTC m=+1839.322425698" watchObservedRunningTime="2026-11-03 23:59:34.22806714 +0000 UTC m=+1839.327667284"
Nov 03 23:59:34.471236 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.471204    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:34.510497 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.510467    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm\" (UniqueName: \"kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm\") pod \"a758da25-0e84-46f4-8664-28e4d347c77a\" (UID: \"a758da25-0e84-46f4-8664-28e4d347c77a\") "
Nov 03 23:59:34.512527 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.512493    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm" pod "a758da25-0e84-46f4-8664-28e4d347c77a" (UID: "a758da25-0e84-46f4-8664-28e4d347c77a"). InnerVolumeSpecName "kube-api-access-c9kxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:34.612819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.612782    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9kxm\" (UniqueName: \"kubernetes.io/projected/a758da25-0e84-46f4-8664-28e4d347c77a-kube-api-access-c9kxm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:34.936219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.936186    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7311-b07ced"]
Nov 03 23:59:34.945201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:34.943357    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7311-b07ced"]
Nov 03 23:59:35.022698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.022664    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a758da25-0e84-46f4-8664-28e4d347c77a" path="/var/lib/kubelet/pods/a758da25-0e84-46f4-8664-28e4d347c77a/volumes"
Nov 03 23:59:35.129843 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.129804    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7311-eac5bb"]
Nov 03 23:59:35.130918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.130888    5074 cpu_manager.go:414] "containerMap: removing container" podUID="900e8995-64b9-4c37-81cd-c0bcd0b4369f" containerName="test-curl-7311-b07ced"
Nov 03 23:59:35.130918 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.130917    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900e8995-64b9-4c37-81cd-c0bcd0b4369f" containerName="test-curl-7311-b07ced"
Nov 03 23:59:35.131175 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.131162    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="900e8995-64b9-4c37-81cd-c0bcd0b4369f" containerName="test-curl-7311-b07ced"
Nov 03 23:59:35.135606 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.135584    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-eac5bb"]
Nov 03 23:59:35.135727 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.135682    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-eac5bb"
Nov 03 23:59:35.216594 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.216562    5074 generic.go:356] "Generic (PLEG): container finished" podID="a758da25-0e84-46f4-8664-28e4d347c77a" containerID="36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634" exitCode=137
Nov 03 23:59:35.216810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.216613    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-bb5953"
Nov 03 23:59:35.216810 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.216661    5074 scope.go:122] "RemoveContainer" containerID="36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634"
Nov 03 23:59:35.223594 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.223566    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtp8\" (UniqueName: \"kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8\") pod \"test-curl-7311-eac5bb\" (UID: \"3e7c28c4-4a93-43b5-8058-52f6382dac77\") " pod="opendatahub/test-curl-7311-eac5bb"
Nov 03 23:59:35.229105 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.229088    5074 scope.go:122] "RemoveContainer" containerID="36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634"
Nov 03 23:59:35.229392 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:35.229374    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634\": container with ID starting with 36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634 not found: ID does not exist" containerID="36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634"
Nov 03 23:59:35.229480 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.229403    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634"} err="failed to get container status \"36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634\": rpc error: code = NotFound desc = could not find container \"36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634\": container with ID starting with 36d6646a4aa3e3717a7ed0573e166f47f46b3220bed3138b1b91835f7e2db634 not found: ID does not exist"
Nov 03 23:59:35.325385 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.325354    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtp8\" (UniqueName: \"kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8\") pod \"test-curl-7311-eac5bb\" (UID: \"3e7c28c4-4a93-43b5-8058-52f6382dac77\") " pod="opendatahub/test-curl-7311-eac5bb"
Nov 03 23:59:35.338986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.338930    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtp8\" (UniqueName: \"kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8\") pod \"test-curl-7311-eac5bb\" (UID: \"3e7c28c4-4a93-43b5-8058-52f6382dac77\") " pod="opendatahub/test-curl-7311-eac5bb"
Nov 03 23:59:35.471734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.471668    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-eac5bb"
Nov 03 23:59:35.789316 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:35.773156    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e7c28c4_4a93_43b5_8058_52f6382dac77.slice/crio-58607c5b88913ba690fe6158ddd8478c3bde9be9198a10dd1716bfe453eea2ed WatchSource:0}: Error finding container 58607c5b88913ba690fe6158ddd8478c3bde9be9198a10dd1716bfe453eea2ed: Status 404 returned error can't find the container with id 58607c5b88913ba690fe6158ddd8478c3bde9be9198a10dd1716bfe453eea2ed
Nov 03 23:59:35.789316 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:35.773611    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-eac5bb"]
Nov 03 23:59:36.230818 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.230781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-eac5bb" event={"ID":"3e7c28c4-4a93-43b5-8058-52f6382dac77","Type":"ContainerStarted","Data":"6a7ce26a7bcc0f84f1636dbfa6aa35669d286952b1530a3cd62145f9e69b3696"}
Nov 03 23:59:36.230954 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.230831    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-eac5bb" event={"ID":"3e7c28c4-4a93-43b5-8058-52f6382dac77","Type":"ContainerStarted","Data":"58607c5b88913ba690fe6158ddd8478c3bde9be9198a10dd1716bfe453eea2ed"}
Nov 03 23:59:36.231091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.231035    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7311-b07ced" podUID="900e8995-64b9-4c37-81cd-c0bcd0b4369f" containerName="test-curl-7311-b07ced" containerID="cri-o://e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f" gracePeriod=2
Nov 03 23:59:36.252577 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.252529    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7311-eac5bb" podStartSLOduration=1.252516453 podStartE2EDuration="1.252516453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:36.247137487 +0000 UTC m=+1841.346737633" watchObservedRunningTime="2026-11-03 23:59:36.252516453 +0000 UTC m=+1841.352116596"
Nov 03 23:59:36.453801 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.453775    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:36.551576 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.551488    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp\" (UniqueName: \"kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp\") pod \"795dbf46-d56e-4bdc-9183-2114f429d2ce\" (UID: \"795dbf46-d56e-4bdc-9183-2114f429d2ce\") "
Nov 03 23:59:36.553621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.553595    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp" pod "795dbf46-d56e-4bdc-9183-2114f429d2ce" (UID: "795dbf46-d56e-4bdc-9183-2114f429d2ce"). InnerVolumeSpecName "kube-api-access-vt7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:36.653823 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:36.653789    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7lp\" (UniqueName: \"kubernetes.io/projected/795dbf46-d56e-4bdc-9183-2114f429d2ce-kube-api-access-vt7lp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:37.019870 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:37.019837    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795dbf46-d56e-4bdc-9183-2114f429d2ce" path="/var/lib/kubelet/pods/795dbf46-d56e-4bdc-9183-2114f429d2ce/volumes"
Nov 03 23:59:37.243349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:37.243309    5074 generic.go:356] "Generic (PLEG): container finished" podID="795dbf46-d56e-4bdc-9183-2114f429d2ce" containerID="ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859" exitCode=137
Nov 03 23:59:37.243349 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:37.243348    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-cf2d4a"
Nov 03 23:59:37.243621 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:37.243403    5074 scope.go:122] "RemoveContainer" containerID="ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859"
Nov 03 23:59:37.254520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:37.254493    5074 scope.go:122] "RemoveContainer" containerID="ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859"
Nov 03 23:59:37.254829 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:37.254806    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859\": container with ID starting with ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859 not found: ID does not exist" containerID="ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859"
Nov 03 23:59:37.254884 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:37.254838    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859"} err="failed to get container status \"ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859\": rpc error: code = NotFound desc = could not find container \"ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859\": container with ID starting with ccdf3efe33b62a7e3ff9c976ba2b6d9314d1ee9158b92195e5ca42f02c5d0859 not found: ID does not exist"
Nov 03 23:59:38.437011 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:38.436979    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:38.487698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:38.487579    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr\" (UniqueName: \"kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr\") pod \"900e8995-64b9-4c37-81cd-c0bcd0b4369f\" (UID: \"900e8995-64b9-4c37-81cd-c0bcd0b4369f\") "
Nov 03 23:59:38.491073 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:38.491026    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr" pod "900e8995-64b9-4c37-81cd-c0bcd0b4369f" (UID: "900e8995-64b9-4c37-81cd-c0bcd0b4369f"). InnerVolumeSpecName "kube-api-access-hpgvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:38.589799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:38.589731    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgvr\" (UniqueName: \"kubernetes.io/projected/900e8995-64b9-4c37-81cd-c0bcd0b4369f-kube-api-access-hpgvr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:38.879670 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:38.879578    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g" podUID="847cbeb0-5f6a-4666-94ec-a78fca0b0839" containerName="main" containerID="cri-o://a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137" gracePeriod=45
Nov 03 23:59:39.022287 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.022244    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900e8995-64b9-4c37-81cd-c0bcd0b4369f" path="/var/lib/kubelet/pods/900e8995-64b9-4c37-81cd-c0bcd0b4369f/volumes"
Nov 03 23:59:39.153971 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.153945    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:59:39.262970 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.262929    5074 generic.go:356] "Generic (PLEG): container finished" podID="900e8995-64b9-4c37-81cd-c0bcd0b4369f" containerID="e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f" exitCode=137
Nov 03 23:59:39.263146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.262993    5074 scope.go:122] "RemoveContainer" containerID="e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f"
Nov 03 23:59:39.263146 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.262991    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-b07ced"
Nov 03 23:59:39.264669 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.264643    5074 generic.go:356] "Generic (PLEG): container finished" podID="847cbeb0-5f6a-4666-94ec-a78fca0b0839" containerID="a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137" exitCode=0
Nov 03 23:59:39.264812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.264680    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"
Nov 03 23:59:39.264812 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.264734    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g" event={"ID":"847cbeb0-5f6a-4666-94ec-a78fca0b0839","Type":"ContainerDied","Data":"a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137"}
Nov 03 23:59:39.264944 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.264834    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g" event={"ID":"847cbeb0-5f6a-4666-94ec-a78fca0b0839","Type":"ContainerDied","Data":"c4baddbc657469f4d79fdb581eabea7b8c50cf0f673fb970fc6d7799f0a4a26e"}
Nov 03 23:59:39.273593 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.273574    5074 scope.go:122] "RemoveContainer" containerID="e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f"
Nov 03 23:59:39.273895 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:39.273862    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f\": container with ID starting with e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f not found: ID does not exist" containerID="e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f"
Nov 03 23:59:39.273990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.273904    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f"} err="failed to get container status \"e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f\": rpc error: code = NotFound desc = could not find container \"e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f\": container with ID starting with e8caac7f6d1d627f1f8f37b2153f7640eff753011f077ba1a999e74189abf73f not found: ID does not exist"
Nov 03 23:59:39.273990 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.273935    5074 scope.go:122] "RemoveContainer" containerID="a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137"
Nov 03 23:59:39.283800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.283782    5074 scope.go:122] "RemoveContainer" containerID="a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137"
Nov 03 23:59:39.284096 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:39.284075    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137\": container with ID starting with a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137 not found: ID does not exist" containerID="a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137"
Nov 03 23:59:39.284158 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.284102    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137"} err="failed to get container status \"a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137\": rpc error: code = NotFound desc = could not find container \"a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137\": container with ID starting with a54d9b64d4329ad9419e8ea6af6e146dd611ce484f4a53c0bcf995411c3ba137 not found: ID does not exist"
Nov 03 23:59:39.309288 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.309212    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache\") pod \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") "
Nov 03 23:59:39.309545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.309376    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l\" (UniqueName: \"kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l\") pod \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") "
Nov 03 23:59:39.309545 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.309514    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs\" (UniqueName: \"kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs\") pod \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") "
Nov 03 23:59:39.309702 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.309610    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home\") pod \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") "
Nov 03 23:59:39.309702 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.309643    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm\") pod \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") "
Nov 03 23:59:39.310075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.309707    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir\") pod \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\" (UID: \"847cbeb0-5f6a-4666-94ec-a78fca0b0839\") "
Nov 03 23:59:39.310933 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.310888    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir" pod "847cbeb0-5f6a-4666-94ec-a78fca0b0839" (UID: "847cbeb0-5f6a-4666-94ec-a78fca0b0839"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:39.311148 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.311120    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache" pod "847cbeb0-5f6a-4666-94ec-a78fca0b0839" (UID: "847cbeb0-5f6a-4666-94ec-a78fca0b0839"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:39.311368 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.311346    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home" pod "847cbeb0-5f6a-4666-94ec-a78fca0b0839" (UID: "847cbeb0-5f6a-4666-94ec-a78fca0b0839"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:39.317325 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.317161    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm" pod "847cbeb0-5f6a-4666-94ec-a78fca0b0839" (UID: "847cbeb0-5f6a-4666-94ec-a78fca0b0839"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:39.330017 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.329985    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs" pod "847cbeb0-5f6a-4666-94ec-a78fca0b0839" (UID: "847cbeb0-5f6a-4666-94ec-a78fca0b0839"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:59:39.333201 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.332986    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l" pod "847cbeb0-5f6a-4666-94ec-a78fca0b0839" (UID: "847cbeb0-5f6a-4666-94ec-a78fca0b0839"). InnerVolumeSpecName "kube-api-access-hmg7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:39.412189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.412094    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:39.412189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.412122    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:39.412189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.412131    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:39.412189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.412139    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:39.412189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.412148    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847cbeb0-5f6a-4666-94ec-a78fca0b0839-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:39.412189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.412158    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmg7l\" (UniqueName: \"kubernetes.io/projected/847cbeb0-5f6a-4666-94ec-a78fca0b0839-kube-api-access-hmg7l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:39.593022 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.592990    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"]
Nov 03 23:59:39.596922 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:39.596895    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/auth-test-model-4331a7-kserve-d7cb57d5-5lw4g"]
Nov 03 23:59:40.034520 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:40.034434    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"]
Nov 03 23:59:40.041036 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:40.041004    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f044c2-zbx7v"]
Nov 03 23:59:41.042051 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:41.042012    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847cbeb0-5f6a-4666-94ec-a78fca0b0839" path="/var/lib/kubelet/pods/847cbeb0-5f6a-4666-94ec-a78fca0b0839/volumes"
Nov 03 23:59:41.043339 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:41.043312    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6dae55-c36d-49c1-bfbb-3545b4e28bb9" path="/var/lib/kubelet/pods/bd6dae55-c36d-49c1-bfbb-3545b4e28bb9/volumes"
Nov 03 23:59:44.168634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.168588    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" podUID="b013efee-c523-4cc4-bd29-6e295477f328" containerName="main" containerID="cri-o://aa40564615c74146f3bbd56d54eb330746f3051a1ba818c1009f1c37fc411a7e" gracePeriod=45
Nov 03 23:59:44.321893 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.321856    5074 generic.go:356] "Generic (PLEG): container finished" podID="b013efee-c523-4cc4-bd29-6e295477f328" containerID="aa40564615c74146f3bbd56d54eb330746f3051a1ba818c1009f1c37fc411a7e" exitCode=0
Nov 03 23:59:44.322095 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.321899    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" event={"ID":"b013efee-c523-4cc4-bd29-6e295477f328","Type":"ContainerDied","Data":"aa40564615c74146f3bbd56d54eb330746f3051a1ba818c1009f1c37fc411a7e"}
Nov 03 23:59:44.329295 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.329254    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" podUID="b013efee-c523-4cc4-bd29-6e295477f328" containerName="main" probeResult="failure" output="Get \"https://10.132.0.189:8000/ready\": dial tcp 10.132.0.189:8000: connect: connection refused"
Nov 03 23:59:44.481800 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.481767    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:59:44.598312 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.598270    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6\" (UniqueName: \"kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6\") pod \"b013efee-c523-4cc4-bd29-6e295477f328\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") "
Nov 03 23:59:44.598479 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.598351    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir\") pod \"b013efee-c523-4cc4-bd29-6e295477f328\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") "
Nov 03 23:59:44.598534 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.598486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm\") pod \"b013efee-c523-4cc4-bd29-6e295477f328\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") "
Nov 03 23:59:44.598582 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.598551    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home\") pod \"b013efee-c523-4cc4-bd29-6e295477f328\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") "
Nov 03 23:59:44.598663 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.598652    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs\" (UniqueName: \"kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs\") pod \"b013efee-c523-4cc4-bd29-6e295477f328\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") "
Nov 03 23:59:44.598989 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.598721    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache\") pod \"b013efee-c523-4cc4-bd29-6e295477f328\" (UID: \"b013efee-c523-4cc4-bd29-6e295477f328\") "
Nov 03 23:59:44.599351 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.599179    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home" pod "b013efee-c523-4cc4-bd29-6e295477f328" (UID: "b013efee-c523-4cc4-bd29-6e295477f328"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:44.599734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.599522    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:44.599842 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.599778    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache" pod "b013efee-c523-4cc4-bd29-6e295477f328" (UID: "b013efee-c523-4cc4-bd29-6e295477f328"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:44.600003 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.599951    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir" pod "b013efee-c523-4cc4-bd29-6e295477f328" (UID: "b013efee-c523-4cc4-bd29-6e295477f328"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:44.602783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.601840    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm" pod "b013efee-c523-4cc4-bd29-6e295477f328" (UID: "b013efee-c523-4cc4-bd29-6e295477f328"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:59:44.602783 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.602263    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6" pod "b013efee-c523-4cc4-bd29-6e295477f328" (UID: "b013efee-c523-4cc4-bd29-6e295477f328"). InnerVolumeSpecName "kube-api-access-tb7t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:44.604439 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.604403    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs" pod "b013efee-c523-4cc4-bd29-6e295477f328" (UID: "b013efee-c523-4cc4-bd29-6e295477f328"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:59:44.701634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.701530    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:44.701634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.701572    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b013efee-c523-4cc4-bd29-6e295477f328-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:44.701634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.701589    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:44.701634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.701606    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7t6\" (UniqueName: \"kubernetes.io/projected/b013efee-c523-4cc4-bd29-6e295477f328-kube-api-access-tb7t6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:44.701634 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:44.701624    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b013efee-c523-4cc4-bd29-6e295477f328-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:45.334072 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:45.333977    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"
Nov 03 23:59:45.334072 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:45.334014    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw" event={"ID":"b013efee-c523-4cc4-bd29-6e295477f328","Type":"ContainerDied","Data":"6bf7dd9c7531462cfd3557d457334febee04c256a7d0d3c0184a96eaf4fccd9b"}
Nov 03 23:59:45.334072 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:45.334068    5074 scope.go:122] "RemoveContainer" containerID="aa40564615c74146f3bbd56d54eb330746f3051a1ba818c1009f1c37fc411a7e"
Nov 03 23:59:45.356778 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:45.356726    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"]
Nov 03 23:59:45.360442 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:45.360420    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/auth-test-model-398158-kserve-867bd6577c-xs5gw"]
Nov 03 23:59:47.021755 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:47.021701    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b013efee-c523-4cc4-bd29-6e295477f328" path="/var/lib/kubelet/pods/b013efee-c523-4cc4-bd29-6e295477f328/volumes"
Nov 03 23:59:51.147177 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.147137    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7314-de203d"]
Nov 03 23:59:51.147868 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.147852    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b013efee-c523-4cc4-bd29-6e295477f328" containerName="main"
Nov 03 23:59:51.147938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.147869    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b013efee-c523-4cc4-bd29-6e295477f328" containerName="main"
Nov 03 23:59:51.147938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.147912    5074 cpu_manager.go:414] "containerMap: removing container" podUID="847cbeb0-5f6a-4666-94ec-a78fca0b0839" containerName="main"
Nov 03 23:59:51.147938 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.147918    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847cbeb0-5f6a-4666-94ec-a78fca0b0839" containerName="main"
Nov 03 23:59:51.148033 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.148027    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="847cbeb0-5f6a-4666-94ec-a78fca0b0839" containerName="main"
Nov 03 23:59:51.148068 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.148040    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b013efee-c523-4cc4-bd29-6e295477f328" containerName="main"
Nov 03 23:59:51.151895 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.151864    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:51.164615 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.156901    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7314-de203d"]
Nov 03 23:59:51.197641 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.197597    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4nfz\" (UniqueName: \"kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz\") pod \"test-curl-7314-de203d\" (UID: \"c72a0150-d1ea-4efd-928b-3b766321cac4\") " pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:51.267785 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.267732    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"]
Nov 03 23:59:51.272356 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.272328    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.277129 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.277103    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:59:51.277409 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.277393    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:59:51.279370 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.279340    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"]
Nov 03 23:59:51.307698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.306520    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.307698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.306723    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4nfz\" (UniqueName: \"kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz\") pod \"test-curl-7314-de203d\" (UID: \"c72a0150-d1ea-4efd-928b-3b766321cac4\") " pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:51.307698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.306784    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgl2m\" (UniqueName: \"kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.307698 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.306870    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.323651 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.323617    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4nfz\" (UniqueName: \"kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz\") pod \"test-curl-7314-de203d\" (UID: \"c72a0150-d1ea-4efd-928b-3b766321cac4\") " pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:51.408561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.408462    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgl2m\" (UniqueName: \"kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.408561 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.408556    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.408860 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.408713    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.410508 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.410480    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.416692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.416659    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.421564 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.421536    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgl2m\" (UniqueName: \"kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m\") pod \"maas-api-revoke-sub-26090e3c-dcv89\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.469236 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.469207    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:51.621345 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.621315    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:51.796762 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.796014    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7314-de203d"]
Nov 03 23:59:51.806219 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:51.806179    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc72a0150_d1ea_4efd_928b_3b766321cac4.slice/crio-3b09ed3162d164dc97f03d4618fb0204a9619aa52b5a27b44c43f1ffd1c60949 WatchSource:0}: Error finding container 3b09ed3162d164dc97f03d4618fb0204a9619aa52b5a27b44c43f1ffd1c60949: Status 404 returned error can't find the container with id 3b09ed3162d164dc97f03d4618fb0204a9619aa52b5a27b44c43f1ffd1c60949
Nov 03 23:59:51.910365 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:51.910335    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"]
Nov 03 23:59:51.910886 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:51.910856    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394d1c2_c8da_4b5d_af3f_d89edb03dd0f.slice/crio-bc918555da7cda899eca2a3c8dc4d14829f433ba80dc06d076651f5b7a48593b WatchSource:0}: Error finding container bc918555da7cda899eca2a3c8dc4d14829f433ba80dc06d076651f5b7a48593b: Status 404 returned error can't find the container with id bc918555da7cda899eca2a3c8dc4d14829f433ba80dc06d076651f5b7a48593b
Nov 03 23:59:52.432799 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:52.432715    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89" event={"ID":"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f","Type":"ContainerStarted","Data":"a2314cd60b8bfe7383db2d108dd03c53e5ffceb13559bcc928190ca4cf666673"}
Nov 03 23:59:52.433296 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:52.432817    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89" event={"ID":"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f","Type":"ContainerStarted","Data":"bc918555da7cda899eca2a3c8dc4d14829f433ba80dc06d076651f5b7a48593b"}
Nov 03 23:59:52.435986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:52.435298    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7314-de203d" event={"ID":"c72a0150-d1ea-4efd-928b-3b766321cac4","Type":"ContainerStarted","Data":"cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e"}
Nov 03 23:59:52.435986 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:52.435346    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7314-de203d" event={"ID":"c72a0150-d1ea-4efd-928b-3b766321cac4","Type":"ContainerStarted","Data":"3b09ed3162d164dc97f03d4618fb0204a9619aa52b5a27b44c43f1ffd1c60949"}
Nov 03 23:59:52.456075 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:52.456010    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89" podStartSLOduration=1.455994225 podStartE2EDuration="1.455994225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:52.451810232 +0000 UTC m=+1857.551410377" watchObservedRunningTime="2026-11-03 23:59:52.455994225 +0000 UTC m=+1857.555594371"
Nov 03 23:59:52.472316 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:52.472249    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7314-de203d" podStartSLOduration=1.472231518 podStartE2EDuration="1.472231518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:52.468176294 +0000 UTC m=+1857.567776440" watchObservedRunningTime="2026-11-03 23:59:52.472231518 +0000 UTC m=+1857.571831664"
Nov 03 23:59:53.177333 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.177298    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7314-de203d"]
Nov 03 23:59:53.183915 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.183884    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7314-de203d"]
Nov 03 23:59:53.447734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.447690    5074 generic.go:356] "Generic (PLEG): container finished" podID="a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" containerID="a2314cd60b8bfe7383db2d108dd03c53e5ffceb13559bcc928190ca4cf666673" exitCode=0
Nov 03 23:59:53.448280 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.447781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89" event={"ID":"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f","Type":"ContainerDied","Data":"a2314cd60b8bfe7383db2d108dd03c53e5ffceb13559bcc928190ca4cf666673"}
Nov 03 23:59:53.458044 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.458015    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"]
Nov 03 23:59:53.458784 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.458768    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" containerName="test-curl-7314-de203d"
Nov 03 23:59:53.458848 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.458786    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" containerName="test-curl-7314-de203d"
Nov 03 23:59:53.458939 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.458929    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" containerName="test-curl-7314-de203d"
Nov 03 23:59:53.462639 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.462621    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.474624 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.470756    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"]
Nov 03 23:59:53.549182 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.549142    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.549350 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.549293    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.549445 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.549397    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdg7\" (UniqueName: \"kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.651334 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.651298    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdg7\" (UniqueName: \"kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.651551 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.651530    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.651653 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.651637    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.652473 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.652442    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.654087 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.654062    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.664448 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.664413    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdg7\" (UniqueName: \"kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7\") pod \"maas-api-revoke-sub-0c72c10c-slw6x\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:53.784405 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:53.784318    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:54.082219 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.082175    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"]
Nov 03 23:59:54.082785 ip-10-0-138-239 kubenswrapper[5074]: W1103 23:59:54.082724    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-d5f38fd5bad87222822b50711c34ac79d4b7273838dcb34f8e1b0e3541c0d348 WatchSource:0}: Error finding container d5f38fd5bad87222822b50711c34ac79d4b7273838dcb34f8e1b0e3541c0d348: Status 404 returned error can't find the container with id d5f38fd5bad87222822b50711c34ac79d4b7273838dcb34f8e1b0e3541c0d348
Nov 03 23:59:54.459315 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.459280    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x" event={"ID":"c6aa3249-0fc9-4c24-93fd-c51e3dccc018","Type":"ContainerStarted","Data":"d5f38fd5bad87222822b50711c34ac79d4b7273838dcb34f8e1b0e3541c0d348"}
Nov 03 23:59:54.459692 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.459442    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7314-de203d" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" containerName="test-curl-7314-de203d" containerID="cri-o://cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e" gracePeriod=2
Nov 03 23:59:54.462823 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:54.462788    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-de203d\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:54.596774 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.596749    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:54.600247 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:54.600014    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-de203d\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:54.668734 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.668697    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca\") pod \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") "
Nov 03 23:59:54.668930 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.668849    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca\") pod \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") "
Nov 03 23:59:54.669176 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.669155    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m\" (UniqueName: \"kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m\") pod \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\" (UID: \"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f\") "
Nov 03 23:59:54.670869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.670838    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca" pod "a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" (UID: "a394d1c2-c8da-4b5d-af3f-d89edb03dd0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:59:54.672876 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.672830    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m" pod "a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" (UID: "a394d1c2-c8da-4b5d-af3f-d89edb03dd0f"). InnerVolumeSpecName "kube-api-access-qgl2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:54.674849 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.674823    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca" pod "a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" (UID: "a394d1c2-c8da-4b5d-af3f-d89edb03dd0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:59:54.783815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.772000    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:54.783815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.772041    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:54.783815 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:54.772069    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgl2m\" (UniqueName: \"kubernetes.io/projected/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f-kube-api-access-qgl2m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:55.021148 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:55.021097    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-de203d\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:55.469690 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:55.469656    5074 generic.go:356] "Generic (PLEG): container finished" podID="c6aa3249-0fc9-4c24-93fd-c51e3dccc018" containerID="c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376" exitCode=0
Nov 03 23:59:55.470165 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:55.469735    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x" event={"ID":"c6aa3249-0fc9-4c24-93fd-c51e3dccc018","Type":"ContainerDied","Data":"c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376"}
Nov 03 23:59:55.471338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:55.471314    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"
Nov 03 23:59:55.471338 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:55.471340    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89" event={"ID":"a394d1c2-c8da-4b5d-af3f-d89edb03dd0f","Type":"ContainerDied","Data":"bc918555da7cda899eca2a3c8dc4d14829f433ba80dc06d076651f5b7a48593b"}
Nov 03 23:59:55.471514 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:55.471385    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc918555da7cda899eca2a3c8dc4d14829f433ba80dc06d076651f5b7a48593b"
Nov 03 23:59:56.655138 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.655106    5074 scope.go:122] "RemoveContainer" containerID="05917ecfb4b2bac518463b56ef9f04490980dd74d519cba00e0fb2adac765ce0"
Nov 03 23:59:56.769628 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.769320    5074 scope.go:122] "RemoveContainer" containerID="ddac08a52153b80e74aae779559fe3b7e2bdcf8917e631128df20eba403ebb65"
Nov 03 23:59:56.774189 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.774162    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:56.782883 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.780137    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:56.785264 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.785243    5074 scope.go:122] "RemoveContainer" containerID="750956ff08df662d28bc5c898a356ec38c11c46ec3e74d348535481fc48959c4"
Nov 03 23:59:56.806792 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.803611    5074 scope.go:122] "RemoveContainer" containerID="95d92d1f4503497bdfa35aa1c79ddbe6cf3c309e2f25d9be0348d673b6f5918a"
Nov 03 23:59:56.824601 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.824560    5074 scope.go:122] "RemoveContainer" containerID="9d05b29da8ace1be533552abff6b9b8f37bada75c26aaf8ac8006101f1b976e6"
Nov 03 23:59:56.839817 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.839652    5074 scope.go:122] "RemoveContainer" containerID="1f597c41fc6d5121909344fadff8b81cf30638b35153bb0f090a1b15223992c8"
Nov 03 23:59:56.855605 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.855583    5074 scope.go:122] "RemoveContainer" containerID="439af23996438adbdf26cb40709a5365ff34ce2d0b263b8308b1f029eac6a90f"
Nov 03 23:59:56.869035 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.868916    5074 scope.go:122] "RemoveContainer" containerID="5283de9449ec696ba05f79c8acc356eff4dbfef17d603cfc2eb2f1bb4464cb89"
Nov 03 23:59:56.879508 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.879482    5074 scope.go:122] "RemoveContainer" containerID="e4f07c9321e288481a5f0b111db768d9ffa89f3b423bd260ced9cae4d0eadb98"
Nov 03 23:59:56.890852 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.890708    5074 scope.go:122] "RemoveContainer" containerID="ef8ae58b087eaa99784c8e47a75518d9be7dee12155a822548b6714a0adc6924"
Nov 03 23:59:56.906447 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.906418    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz\" (UniqueName: \"kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz\") pod \"c72a0150-d1ea-4efd-928b-3b766321cac4\" (UID: \"c72a0150-d1ea-4efd-928b-3b766321cac4\") "
Nov 03 23:59:56.906588 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.906571    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca\") pod \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") "
Nov 03 23:59:56.906683 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.906668    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7\" (UniqueName: \"kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7\") pod \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") "
Nov 03 23:59:56.906819 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.906705    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca\") pod \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\" (UID: \"c6aa3249-0fc9-4c24-93fd-c51e3dccc018\") "
Nov 03 23:59:56.907191 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.907155    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca" pod "c6aa3249-0fc9-4c24-93fd-c51e3dccc018" (UID: "c6aa3249-0fc9-4c24-93fd-c51e3dccc018"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:59:56.907571 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.907549    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:56.909636 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.909590    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca" pod "c6aa3249-0fc9-4c24-93fd-c51e3dccc018" (UID: "c6aa3249-0fc9-4c24-93fd-c51e3dccc018"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:59:56.909831 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.909779    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7" pod "c6aa3249-0fc9-4c24-93fd-c51e3dccc018" (UID: "c6aa3249-0fc9-4c24-93fd-c51e3dccc018"). InnerVolumeSpecName "kube-api-access-hcdg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:56.911167 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:56.911143    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz" pod "c72a0150-d1ea-4efd-928b-3b766321cac4" (UID: "c72a0150-d1ea-4efd-928b-3b766321cac4"). InnerVolumeSpecName "kube-api-access-j4nfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:59:57.009091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.009057    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdg7\" (UniqueName: \"kubernetes.io/projected/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-kube-api-access-hcdg7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:57.009091 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.009091    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aa3249-0fc9-4c24-93fd-c51e3dccc018-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:57.009268 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.009106    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4nfz\" (UniqueName: \"kubernetes.io/projected/c72a0150-d1ea-4efd-928b-3b766321cac4-kube-api-access-j4nfz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:59:57.020311 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.020281    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72a0150-d1ea-4efd-928b-3b766321cac4" path="/var/lib/kubelet/pods/c72a0150-d1ea-4efd-928b-3b766321cac4/volumes"
Nov 03 23:59:57.495402 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.495367    5074 generic.go:356] "Generic (PLEG): container finished" podID="c72a0150-d1ea-4efd-928b-3b766321cac4" containerID="cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e" exitCode=137
Nov 03 23:59:57.495599 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.495451    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-de203d"
Nov 03 23:59:57.495599 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.495474    5074 scope.go:122] "RemoveContainer" containerID="cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e"
Nov 03 23:59:57.497881 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.497860    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x" event={"ID":"c6aa3249-0fc9-4c24-93fd-c51e3dccc018","Type":"ContainerDied","Data":"d5f38fd5bad87222822b50711c34ac79d4b7273838dcb34f8e1b0e3541c0d348"}
Nov 03 23:59:57.497999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.497905    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f38fd5bad87222822b50711c34ac79d4b7273838dcb34f8e1b0e3541c0d348"
Nov 03 23:59:57.497999 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.497970    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"
Nov 03 23:59:57.511344 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.511318    5074 scope.go:122] "RemoveContainer" containerID="cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e"
Nov 03 23:59:57.511869 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:57.511695    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e\": container with ID starting with cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e not found: ID does not exist" containerID="cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e"
Nov 03 23:59:57.511869 ip-10-0-138-239 kubenswrapper[5074]: I1103 23:59:57.511731    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e"} err="failed to get container status \"cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e\": rpc error: code = NotFound desc = could not find container \"cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e\": container with ID starting with cc9af6553c0384b3e8247af89702db64394d650a1e5325d4d0698edbdf5bbc1e not found: ID does not exist"
Nov 03 23:59:58.028617 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:58.028577    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 03 23:59:59.697613 ip-10-0-138-239 kubenswrapper[5074]: E1103 23:59:59.697573    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:00.050638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.050533    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"]
Nov 04 00:00:00.054379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.054349    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c12e230-k9s9s"]
Nov 04 00:00:00.142563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.142521    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"]
Nov 04 00:00:00.143699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.143660    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" containerName="revoke-keys"
Nov 04 00:00:00.143856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.143701    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" containerName="revoke-keys"
Nov 04 00:00:00.143856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.143729    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c6aa3249-0fc9-4c24-93fd-c51e3dccc018" containerName="revoke-keys"
Nov 04 00:00:00.143856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.143779    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6aa3249-0fc9-4c24-93fd-c51e3dccc018" containerName="revoke-keys"
Nov 04 00:00:00.144022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.143982    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6aa3249-0fc9-4c24-93fd-c51e3dccc018" containerName="revoke-keys"
Nov 04 00:00:00.144022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.144009    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" containerName="revoke-keys"
Nov 04 00:00:00.151158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.151131    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"]
Nov 04 00:00:00.151306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.151277    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.156630 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.156600    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"]
Nov 04 00:00:00.156811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.156785    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.162076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.162050    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"]
Nov 04 00:00:00.162181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.162131    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.180996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.166934    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:00:00.180996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.167778    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"]
Nov 04 00:00:00.180996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.167877    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.180996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.169570    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"]
Nov 04 00:00:00.180996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.179692    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"]
Nov 04 00:00:00.192900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.192863    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"]
Nov 04 00:00:00.201056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.197132    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"]
Nov 04 00:00:00.217347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.217319    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.228651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.228620    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"]
Nov 04 00:00:00.235590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.235559    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.241654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.238843    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"]
Nov 04 00:00:00.257508 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:00.256882    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:00.260841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.260806    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.261066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.261047    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.261156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.261093    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhh9x\" (UniqueName: \"kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.261156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.261133    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7z2\" (UniqueName: \"kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.261283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.261177    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmcf\" (UniqueName: \"kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.261283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.261234    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.266698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.262584    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"]
Nov 04 00:00:00.266698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.263833    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.357585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.357549    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"]
Nov 04 00:00:00.357769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.357658    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.357853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.357836    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"]
Nov 04 00:00:00.357918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.357875    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"]
Nov 04 00:00:00.357918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.357901    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"]
Nov 04 00:00:00.363524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363104    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.363524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363166    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.363524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363233    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.363524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363304    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.363524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363362    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.363524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363419    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97ds\" (UniqueName: \"kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363575    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqdf\" (UniqueName: \"kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363646    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkg2\" (UniqueName: \"kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363678    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363724    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhh9x\" (UniqueName: \"kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363817    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7z2\" (UniqueName: \"kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363869    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmcf\" (UniqueName: \"kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363905    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4rc\" (UniqueName: \"kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.364069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.363974    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.364819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.364797    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"]
Nov 04 00:00:00.365655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.365633    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.366695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.366665    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.369814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.364823    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.370863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.370839    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.375122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.375099    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29895840-b78h6"]
Nov 04 00:00:00.375701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.375683    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.381456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.381408    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"]
Nov 04 00:00:00.381560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.381492    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"]
Nov 04 00:00:00.381796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.381775    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7z2\" (UniqueName: \"kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.381923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.381892    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.382612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.382588    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhh9x\" (UniqueName: \"kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.386142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.386102    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmcf\" (UniqueName: \"kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.386641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.386617    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-g89f7\""
Nov 04 00:00:00.387191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.387171    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"]
Nov 04 00:00:00.387348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.387203    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"]
Nov 04 00:00:00.387348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.387273    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.389209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.389188    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29895840-b78h6"]
Nov 04 00:00:00.389916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.389884    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Nov 04 00:00:00.395306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.395284    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"]
Nov 04 00:00:00.464967 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.464931    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97ds\" (UniqueName: \"kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.465105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465021    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4fz\" (UniqueName: \"kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.465105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465068    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqdf\" (UniqueName: \"kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.465243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465107    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkg2\" (UniqueName: \"kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.465243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465141    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.465243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465185    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4rc\" (UniqueName: \"kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.465243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465232    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.465462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465264    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.465462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465299    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.465462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465358    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.465462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465375    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnb2\" (UniqueName: \"kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.465462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.465399    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.466076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.466050    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.467411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.467385    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.467518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.467500    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.467665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.467642    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.479938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.479914    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97ds\" (UniqueName: \"kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.480281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.480256    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4rc\" (UniqueName: \"kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.480542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.480525    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqdf\" (UniqueName: \"kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.481191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.481169    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkg2\" (UniqueName: \"kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.485789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.485763    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7311-eac5bb"]
Nov 04 00:00:00.486519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.485964    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7311-eac5bb" podUID="3e7c28c4-4a93-43b5-8058-52f6382dac77" containerName="test-curl-7311-eac5bb" containerID="cri-o://6a7ce26a7bcc0f84f1636dbfa6aa35669d286952b1530a3cd62145f9e69b3696" gracePeriod=2
Nov 04 00:00:00.489422 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:00.489392    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-eac5bb\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="3e7c28c4-4a93-43b5-8058-52f6382dac77" pod="opendatahub/test-curl-7311-eac5bb"
Nov 04 00:00:00.491532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.491511    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7311-eac5bb"]
Nov 04 00:00:00.539761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.539034    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:00.566708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.566679    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4fz\" (UniqueName: \"kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.566883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.566730    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xnrx\" (UniqueName: \"kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.566883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.566859    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.567011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.566926    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca\") pod \"image-pruner-29895840-b78h6\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") " pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.567011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.566998    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.567296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.567273    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:00.567697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.567666    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxr8\" (UniqueName: \"kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.567794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.567783    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v25kr\" (UniqueName: \"kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr\") pod \"image-pruner-29895840-b78h6\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") " pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.567854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.567788    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.567920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.567892    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.568062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.567989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnb2\" (UniqueName: \"kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.568062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.568048    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.568641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.568606    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.581062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.581039    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4fz\" (UniqueName: \"kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.581809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.581787    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnb2\" (UniqueName: \"kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.583850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.583828    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:00.662514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.662250    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:00.664912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.664880    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.670711    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xnrx\" (UniqueName: \"kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.670786    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.670838    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca\") pod \"image-pruner-29895840-b78h6\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") " pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.670919    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxr8\" (UniqueName: \"kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.670997    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v25kr\" (UniqueName: \"kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr\") pod \"image-pruner-29895840-b78h6\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") " pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.671655    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.673259    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca\") pod \"image-pruner-29895840-b78h6\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") " pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.674276    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.675115    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.676108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.675588    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:00.677177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.677152    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.688071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.687524    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xnrx\" (UniqueName: \"kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.714609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.714548    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v25kr\" (UniqueName: \"kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr\") pod \"image-pruner-29895840-b78h6\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") " pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.721010 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.715426    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxr8\" (UniqueName: \"kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:00.726578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.725195    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:00.739208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.738984    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:00.751676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.751455    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:00.756210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.756011    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:00.763494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:00.763339    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:01.067282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.066730    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f1ed1c-668e-4ee4-96c7-022057322e96" path="/var/lib/kubelet/pods/49f1ed1c-668e-4ee4-96c7-022057322e96/volumes"
Nov 04 00:00:01.093521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.093008    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"]
Nov 04 00:00:01.095374 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:01.095348    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc0c6f8_14dd_404b_af9c_2651eaf5cf75.slice/crio-317e96a48971bc6518f9d3de215598d29ef6c4588556166eca0191d247bed792 WatchSource:0}: Error finding container 317e96a48971bc6518f9d3de215598d29ef6c4588556166eca0191d247bed792: Status 404 returned error can't find the container with id 317e96a48971bc6518f9d3de215598d29ef6c4588556166eca0191d247bed792
Nov 04 00:00:01.117643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.117576    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"]
Nov 04 00:00:01.290963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.290853    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"]
Nov 04 00:00:01.299475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.299380    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"]
Nov 04 00:00:01.300068 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:01.300018    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c21e35_deb3_4c06_ab24_9c372d4f1ba0.slice/crio-125381af6f64ce45dd15be1b0a1ce61a8e19a2e8bd3d7599ce6df492e3b2a77e WatchSource:0}: Error finding container 125381af6f64ce45dd15be1b0a1ce61a8e19a2e8bd3d7599ce6df492e3b2a77e: Status 404 returned error can't find the container with id 125381af6f64ce45dd15be1b0a1ce61a8e19a2e8bd3d7599ce6df492e3b2a77e
Nov 04 00:00:01.302651 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:01.302618    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ce3fbf_89e5_4766_a524_968f410af728.slice/crio-10b4fb106e9d6a9c62b1c2bb36fd1e35b0fd6b3878f8479e68106e2a32a3b8a7 WatchSource:0}: Error finding container 10b4fb106e9d6a9c62b1c2bb36fd1e35b0fd6b3878f8479e68106e2a32a3b8a7: Status 404 returned error can't find the container with id 10b4fb106e9d6a9c62b1c2bb36fd1e35b0fd6b3878f8479e68106e2a32a3b8a7
Nov 04 00:00:01.307126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.307071    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"]
Nov 04 00:00:01.393778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.393726    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"]
Nov 04 00:00:01.419278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.419017    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"]
Nov 04 00:00:01.521902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.521723    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"]
Nov 04 00:00:01.537933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.537901    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"]
Nov 04 00:00:01.550041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.547174    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"]
Nov 04 00:00:01.586484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.585886    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerStarted","Data":"13b64585f75fef5ec1ccacde52c3fc85137a7eabd777be25ee0321af1ea4be23"}
Nov 04 00:00:01.623593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.600428    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerStarted","Data":"125381af6f64ce45dd15be1b0a1ce61a8e19a2e8bd3d7599ce6df492e3b2a77e"}
Nov 04 00:00:01.623593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.616702    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"]
Nov 04 00:00:01.652896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.652857    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerStarted","Data":"991dc4c008b91353047f5b83db129dae926ec00f37b2e020e19c3d8c19fb736a"}
Nov 04 00:00:01.654732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.654549    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29895840-b78h6"]
Nov 04 00:00:01.655763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.655381    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerStarted","Data":"7b4ed2c2c069ddc4c2acc76aab6000f5688b892cd88ce102d8ea24d4bf3a0535"}
Nov 04 00:00:01.657472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.657446    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerStarted","Data":"10b4fb106e9d6a9c62b1c2bb36fd1e35b0fd6b3878f8479e68106e2a32a3b8a7"}
Nov 04 00:00:01.659963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.659513    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerStarted","Data":"a9624979471b1b79031e70d822e0ca17983b855b5065c5d206acc691158a1f84"}
Nov 04 00:00:01.660784 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:01.660730    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373f1058_e636_45d6_a53b_cdef5a009136.slice/crio-5d438a8fa7a59008d919d400a2474ef47a4b41067cc4ebc7d7d1b3b5f8bf9009 WatchSource:0}: Error finding container 5d438a8fa7a59008d919d400a2474ef47a4b41067cc4ebc7d7d1b3b5f8bf9009: Status 404 returned error can't find the container with id 5d438a8fa7a59008d919d400a2474ef47a4b41067cc4ebc7d7d1b3b5f8bf9009
Nov 04 00:00:01.663314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.662506    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerStarted","Data":"9002f7066c93f19a91ac97316c5b25a6263609e1785f907dbd933fafded11a5f"}
Nov 04 00:00:01.664515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:01.664476    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerStarted","Data":"317e96a48971bc6518f9d3de215598d29ef6c4588556166eca0191d247bed792"}
Nov 04 00:00:02.643519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.643489    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7311-4cb031"]
Nov 04 00:00:02.644602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.644575    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3e7c28c4-4a93-43b5-8058-52f6382dac77" containerName="test-curl-7311-eac5bb"
Nov 04 00:00:02.644602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.644602    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7c28c4-4a93-43b5-8058-52f6382dac77" containerName="test-curl-7311-eac5bb"
Nov 04 00:00:02.644887 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.644865    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7c28c4-4a93-43b5-8058-52f6382dac77" containerName="test-curl-7311-eac5bb"
Nov 04 00:00:02.652786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.652071    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:02.656733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.656701    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-4cb031"]
Nov 04 00:00:02.729991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.714080    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerStarted","Data":"0a79a26b3073ca9bea586358d7952f8210f32e37c86992ca7b358c555a36bae4"}
Nov 04 00:00:02.750644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.743494    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcbx\" (UniqueName: \"kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx\") pod \"test-curl-7311-4cb031\" (UID: \"9c00755e-e4a6-4679-9e54-68f85af58467\") " pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:02.765431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.765362    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" podStartSLOduration=2.765340681 podStartE2EDuration="2.765340681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.763149494 +0000 UTC m=+1867.862749662" watchObservedRunningTime="2026-11-04 00:00:02.765340681 +0000 UTC m=+1867.864940825"
Nov 04 00:00:02.779474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.779437    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerStarted","Data":"8c3ad56731eeef47ae41f7f1e1283b7181df1005ce03950b357c388e0672d978"}
Nov 04 00:00:02.786775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.785990    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerStarted","Data":"31a4df28e18f53030819d334c1d4f93bf2f82a0326c6cbcc6876687ee3205353"}
Nov 04 00:00:02.786775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.786415    5074 scope.go:122] "RemoveContainer" containerID="31a4df28e18f53030819d334c1d4f93bf2f82a0326c6cbcc6876687ee3205353"
Nov 04 00:00:02.799687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.799657    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerStarted","Data":"4bf2aa1883e386f91828d9ca4b93161620a7e168a9f3cf61dd941f3670f22d63"}
Nov 04 00:00:02.799897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.799878    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerStarted","Data":"0c24e90cb582c33226af675ded2aecbb003448d465e0446bc2634d5b9da830d3"}
Nov 04 00:00:02.812846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.811594    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerStarted","Data":"89746a56e838455de1f97c3adefdd0f18860fede007c4ff8cd535cc92ae5914f"}
Nov 04 00:00:02.815017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.814393    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerStarted","Data":"978fc6c4865c9291b267d5a374305256219329ee44e911ca4a96b33309f7a7aa"}
Nov 04 00:00:02.817007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.816911    5074 generic.go:356] "Generic (PLEG): container finished" podID="3e7c28c4-4a93-43b5-8058-52f6382dac77" containerID="6a7ce26a7bcc0f84f1636dbfa6aa35669d286952b1530a3cd62145f9e69b3696" exitCode=137
Nov 04 00:00:02.817007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.816950    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" podStartSLOduration=2.816935884 podStartE2EDuration="2.816935884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.800974776 +0000 UTC m=+1867.900574922" watchObservedRunningTime="2026-11-04 00:00:02.816935884 +0000 UTC m=+1867.916536028"
Nov 04 00:00:02.819734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.818807    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerStarted","Data":"adece17bfcb0d04f884d9c327f22a4c907e437b921b341ced42ef8e158092ffd"}
Nov 04 00:00:02.820146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.820125    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerStarted","Data":"d10ca78c6bd48b8b3b3d204b6b36723419e9707e68cbda0628a9c9f35b8a325d"}
Nov 04 00:00:02.820948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.820808    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29895840-b78h6" event={"ID":"373f1058-e636-45d6-a53b-cdef5a009136","Type":"ContainerStarted","Data":"eecb49d781cd7dd5e14c94a3aa93d1fe7b93fb97e1876515a85b79475e4c706c"}
Nov 04 00:00:02.820948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.820839    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29895840-b78h6" event={"ID":"373f1058-e636-45d6-a53b-cdef5a009136","Type":"ContainerStarted","Data":"5d438a8fa7a59008d919d400a2474ef47a4b41067cc4ebc7d7d1b3b5f8bf9009"}
Nov 04 00:00:02.822637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.822614    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerStarted","Data":"65aa3399a1c4f7b8b8c6bfc021ed6968187eb736ee65d29bc3628b63c9d9cfef"}
Nov 04 00:00:02.822725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.822659    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerStarted","Data":"d8adcab579157e3f6c7121dbe9979dd1692639501cdd03ffbb3d2ea83db546ab"}
Nov 04 00:00:02.825402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.825072    5074 scope.go:122] "RemoveContainer" containerID="c83f237d3a4ea867058366e45d108d06aa80f067e5e4ba10ae812ba35b6fbf95"
Nov 04 00:00:02.826168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.826101    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerDied","Data":"c83f237d3a4ea867058366e45d108d06aa80f067e5e4ba10ae812ba35b6fbf95"}
Nov 04 00:00:02.826648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.824731    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerID="c83f237d3a4ea867058366e45d108d06aa80f067e5e4ba10ae812ba35b6fbf95" exitCode=60
Nov 04 00:00:02.833780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.833481    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerStarted","Data":"52b8073b7a60238728ed52b70dfe0736bcec4eebab1de64f7d5f1049e2038e04"}
Nov 04 00:00:02.846678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.846617    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" podStartSLOduration=2.846600553 podStartE2EDuration="2.846600553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.843918677 +0000 UTC m=+1867.943518828" watchObservedRunningTime="2026-11-04 00:00:02.846600553 +0000 UTC m=+1867.946200700"
Nov 04 00:00:02.853928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.847723    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcbx\" (UniqueName: \"kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx\") pod \"test-curl-7311-4cb031\" (UID: \"9c00755e-e4a6-4679-9e54-68f85af58467\") " pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:02.863894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.863126    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerStarted","Data":"aa9ab6c127725723852fe515cea48b4951c48382019497510159355e45f23dff"}
Nov 04 00:00:02.868060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.868039    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcbx\" (UniqueName: \"kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx\") pod \"test-curl-7311-4cb031\" (UID: \"9c00755e-e4a6-4679-9e54-68f85af58467\") " pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:02.868884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.867535    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" podStartSLOduration=2.8675195220000003 podStartE2EDuration="2.867519522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.865055974 +0000 UTC m=+1867.964656119" watchObservedRunningTime="2026-11-04 00:00:02.867519522 +0000 UTC m=+1867.967119666"
Nov 04 00:00:02.942060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.942003    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" podStartSLOduration=2.9419849080000002 podStartE2EDuration="2.941984908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.938917564 +0000 UTC m=+1868.038517708" watchObservedRunningTime="2026-11-04 00:00:02.941984908 +0000 UTC m=+1868.041585052"
Nov 04 00:00:02.942788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.942707    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" podStartSLOduration=2.942697759 podStartE2EDuration="2.942697759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.893775763 +0000 UTC m=+1867.993375908" watchObservedRunningTime="2026-11-04 00:00:02.942697759 +0000 UTC m=+1868.042297904"
Nov 04 00:00:02.987175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.987115    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" podStartSLOduration=2.987096911 podStartE2EDuration="2.987096911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.979887507 +0000 UTC m=+1868.079487652" watchObservedRunningTime="2026-11-04 00:00:02.987096911 +0000 UTC m=+1868.086697060"
Nov 04 00:00:02.994520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:02.994494    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-eac5bb"
Nov 04 00:00:03.005616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.005284    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" podStartSLOduration=3.005269014 podStartE2EDuration="3.005269014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.00502173 +0000 UTC m=+1868.104621875" watchObservedRunningTime="2026-11-04 00:00:03.005269014 +0000 UTC m=+1868.104869162"
Nov 04 00:00:03.034604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.034559    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29895840-b78h6" podStartSLOduration=3.034546571 podStartE2EDuration="3.034546571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.033692451 +0000 UTC m=+1868.133292598" watchObservedRunningTime="2026-11-04 00:00:03.034546571 +0000 UTC m=+1868.134146716"
Nov 04 00:00:03.057476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.057430    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" podStartSLOduration=3.057415909 podStartE2EDuration="3.057415909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.054990768 +0000 UTC m=+1868.154590912" watchObservedRunningTime="2026-11-04 00:00:03.057415909 +0000 UTC m=+1868.157016053"
Nov 04 00:00:03.057819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.057798    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8\" (UniqueName: \"kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8\") pod \"3e7c28c4-4a93-43b5-8058-52f6382dac77\" (UID: \"3e7c28c4-4a93-43b5-8058-52f6382dac77\") "
Nov 04 00:00:03.101147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.101076    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8" pod "3e7c28c4-4a93-43b5-8058-52f6382dac77" (UID: "3e7c28c4-4a93-43b5-8058-52f6382dac77"). InnerVolumeSpecName "kube-api-access-zqtp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:03.159647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.159616    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtp8\" (UniqueName: \"kubernetes.io/projected/3e7c28c4-4a93-43b5-8058-52f6382dac77-kube-api-access-zqtp8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:03.164843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.164818    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:03.476288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.476262    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-4cb031"]
Nov 04 00:00:03.501882 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:03.501851    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c00755e_e4a6_4679_9e54_68f85af58467.slice/crio-878e9fe3b03d3758dad238da00d6885ebf1c7b7d60d4cf5ec7708ee68cfc85f3 WatchSource:0}: Error finding container 878e9fe3b03d3758dad238da00d6885ebf1c7b7d60d4cf5ec7708ee68cfc85f3: Status 404 returned error can't find the container with id 878e9fe3b03d3758dad238da00d6885ebf1c7b7d60d4cf5ec7708ee68cfc85f3
Nov 04 00:00:03.752693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.752653    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"]
Nov 04 00:00:03.758457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.758381    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.760384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.760353    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"]
Nov 04 00:00:03.763153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.763129    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:00:03.867982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.867945    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.867982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.867989    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r752k\" (UniqueName: \"kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.868230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.868164    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.874182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.874148    5074 generic.go:356] "Generic (PLEG): container finished" podID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerID="0a79a26b3073ca9bea586358d7952f8210f32e37c86992ca7b358c555a36bae4" exitCode=60
Nov 04 00:00:03.874325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.874239    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerDied","Data":"0a79a26b3073ca9bea586358d7952f8210f32e37c86992ca7b358c555a36bae4"}
Nov 04 00:00:03.874544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.874528    5074 scope.go:122] "RemoveContainer" containerID="0a79a26b3073ca9bea586358d7952f8210f32e37c86992ca7b358c555a36bae4"
Nov 04 00:00:03.876010 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.875983    5074 generic.go:356] "Generic (PLEG): container finished" podID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerID="8c3ad56731eeef47ae41f7f1e1283b7181df1005ce03950b357c388e0672d978" exitCode=60
Nov 04 00:00:03.876120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.876062    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerDied","Data":"8c3ad56731eeef47ae41f7f1e1283b7181df1005ce03950b357c388e0672d978"}
Nov 04 00:00:03.876412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.876308    5074 scope.go:122] "RemoveContainer" containerID="8c3ad56731eeef47ae41f7f1e1283b7181df1005ce03950b357c388e0672d978"
Nov 04 00:00:03.878227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.878205    5074 generic.go:356] "Generic (PLEG): container finished" podID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerID="31a4df28e18f53030819d334c1d4f93bf2f82a0326c6cbcc6876687ee3205353" exitCode=60
Nov 04 00:00:03.878336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.878284    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerDied","Data":"31a4df28e18f53030819d334c1d4f93bf2f82a0326c6cbcc6876687ee3205353"}
Nov 04 00:00:03.878336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.878331    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerStarted","Data":"3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd"}
Nov 04 00:00:03.883902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.883876    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3fab351-0452-48f0-9d68-260c672a0e85" containerID="4bf2aa1883e386f91828d9ca4b93161620a7e168a9f3cf61dd941f3670f22d63" exitCode=60
Nov 04 00:00:03.883981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.883961    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerDied","Data":"4bf2aa1883e386f91828d9ca4b93161620a7e168a9f3cf61dd941f3670f22d63"}
Nov 04 00:00:03.884161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.884148    5074 scope.go:122] "RemoveContainer" containerID="4bf2aa1883e386f91828d9ca4b93161620a7e168a9f3cf61dd941f3670f22d63"
Nov 04 00:00:03.885625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.885608    5074 generic.go:356] "Generic (PLEG): container finished" podID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerID="89746a56e838455de1f97c3adefdd0f18860fede007c4ff8cd535cc92ae5914f" exitCode=60
Nov 04 00:00:03.885715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.885687    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerDied","Data":"89746a56e838455de1f97c3adefdd0f18860fede007c4ff8cd535cc92ae5914f"}
Nov 04 00:00:03.885900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.885882    5074 scope.go:122] "RemoveContainer" containerID="89746a56e838455de1f97c3adefdd0f18860fede007c4ff8cd535cc92ae5914f"
Nov 04 00:00:03.887450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.887429    5074 generic.go:356] "Generic (PLEG): container finished" podID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerID="978fc6c4865c9291b267d5a374305256219329ee44e911ca4a96b33309f7a7aa" exitCode=60
Nov 04 00:00:03.887518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.887503    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerDied","Data":"978fc6c4865c9291b267d5a374305256219329ee44e911ca4a96b33309f7a7aa"}
Nov 04 00:00:03.887690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.887675    5074 scope.go:122] "RemoveContainer" containerID="978fc6c4865c9291b267d5a374305256219329ee44e911ca4a96b33309f7a7aa"
Nov 04 00:00:03.889151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.889122    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-4cb031" event={"ID":"9c00755e-e4a6-4679-9e54-68f85af58467","Type":"ContainerStarted","Data":"cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949"}
Nov 04 00:00:03.889242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.889151    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-4cb031" event={"ID":"9c00755e-e4a6-4679-9e54-68f85af58467","Type":"ContainerStarted","Data":"878e9fe3b03d3758dad238da00d6885ebf1c7b7d60d4cf5ec7708ee68cfc85f3"}
Nov 04 00:00:03.890651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.890635    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-eac5bb"
Nov 04 00:00:03.890775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.890655    5074 scope.go:122] "RemoveContainer" containerID="6a7ce26a7bcc0f84f1636dbfa6aa35669d286952b1530a3cd62145f9e69b3696"
Nov 04 00:00:03.892808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.892785    5074 generic.go:356] "Generic (PLEG): container finished" podID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerID="adece17bfcb0d04f884d9c327f22a4c907e437b921b341ced42ef8e158092ffd" exitCode=60
Nov 04 00:00:03.894863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.893955    5074 scope.go:122] "RemoveContainer" containerID="adece17bfcb0d04f884d9c327f22a4c907e437b921b341ced42ef8e158092ffd"
Nov 04 00:00:03.894989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.893981    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerDied","Data":"adece17bfcb0d04f884d9c327f22a4c907e437b921b341ced42ef8e158092ffd"}
Nov 04 00:00:03.915297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.915265    5074 generic.go:356] "Generic (PLEG): container finished" podID="373f1058-e636-45d6-a53b-cdef5a009136" containerID="eecb49d781cd7dd5e14c94a3aa93d1fe7b93fb97e1876515a85b79475e4c706c" exitCode=0
Nov 04 00:00:03.915414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.915353    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29895840-b78h6" event={"ID":"373f1058-e636-45d6-a53b-cdef5a009136","Type":"ContainerDied","Data":"eecb49d781cd7dd5e14c94a3aa93d1fe7b93fb97e1876515a85b79475e4c706c"}
Nov 04 00:00:03.930063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.930032    5074 generic.go:356] "Generic (PLEG): container finished" podID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerID="65aa3399a1c4f7b8b8c6bfc021ed6968187eb736ee65d29bc3628b63c9d9cfef" exitCode=60
Nov 04 00:00:03.930162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.930132    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerDied","Data":"65aa3399a1c4f7b8b8c6bfc021ed6968187eb736ee65d29bc3628b63c9d9cfef"}
Nov 04 00:00:03.930549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.930531    5074 scope.go:122] "RemoveContainer" containerID="65aa3399a1c4f7b8b8c6bfc021ed6968187eb736ee65d29bc3628b63c9d9cfef"
Nov 04 00:00:03.933189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.933163    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerStarted","Data":"38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642"}
Nov 04 00:00:03.943932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.943909    5074 generic.go:356] "Generic (PLEG): container finished" podID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerID="52b8073b7a60238728ed52b70dfe0736bcec4eebab1de64f7d5f1049e2038e04" exitCode=60
Nov 04 00:00:03.944429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.944387    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerDied","Data":"52b8073b7a60238728ed52b70dfe0736bcec4eebab1de64f7d5f1049e2038e04"}
Nov 04 00:00:03.945012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.944994    5074 scope.go:122] "RemoveContainer" containerID="52b8073b7a60238728ed52b70dfe0736bcec4eebab1de64f7d5f1049e2038e04"
Nov 04 00:00:03.947644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.947590    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" podStartSLOduration=3.947573635 podStartE2EDuration="3.947573635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.940431953 +0000 UTC m=+1869.040032097" watchObservedRunningTime="2026-11-04 00:00:03.947573635 +0000 UTC m=+1869.047173782"
Nov 04 00:00:03.952192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.952168    5074 generic.go:356] "Generic (PLEG): container finished" podID="94ce3fbf-89e5-4766-a524-968f410af728" containerID="aa9ab6c127725723852fe515cea48b4951c48382019497510159355e45f23dff" exitCode=60
Nov 04 00:00:03.952292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.952202    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerDied","Data":"aa9ab6c127725723852fe515cea48b4951c48382019497510159355e45f23dff"}
Nov 04 00:00:03.952462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.952445    5074 scope.go:122] "RemoveContainer" containerID="aa9ab6c127725723852fe515cea48b4951c48382019497510159355e45f23dff"
Nov 04 00:00:03.962979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.962931    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7311-4cb031" podStartSLOduration=1.962916667 podStartE2EDuration="1.962916667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.960379463 +0000 UTC m=+1869.059979610" watchObservedRunningTime="2026-11-04 00:00:03.962916667 +0000 UTC m=+1869.062516813"
Nov 04 00:00:03.969824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.969796    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.970166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.970142    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.970307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.970189    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r752k\" (UniqueName: \"kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.971036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.971012    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.977332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.977312    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:03.987649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:03.987616    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r752k\" (UniqueName: \"kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k\") pod \"maas-api-revoke-sub-194475f0-7k7rm\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:04.035166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.035130    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"]
Nov 04 00:00:04.039847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.039824    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10a7c73-gh9rt"]
Nov 04 00:00:04.108698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.108667    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:04.130576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.130515    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" podStartSLOduration=4.130496413 podStartE2EDuration="4.130496413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:04.126310608 +0000 UTC m=+1869.225910752" watchObservedRunningTime="2026-11-04 00:00:04.130496413 +0000 UTC m=+1869.230096558"
Nov 04 00:00:04.522145 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:04.522112    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2e92fa_4fbe_44b8_a3f6_b700da748344.slice/crio-5b068cf7ea1e68661ad54a9ff3125c2e976367e5dcb9e2ceb4494c406922d3b5 WatchSource:0}: Error finding container 5b068cf7ea1e68661ad54a9ff3125c2e976367e5dcb9e2ceb4494c406922d3b5: Status 404 returned error can't find the container with id 5b068cf7ea1e68661ad54a9ff3125c2e976367e5dcb9e2ceb4494c406922d3b5
Nov 04 00:00:04.524596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.524568    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"]
Nov 04 00:00:04.716989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.716909    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7311-4cb031"]
Nov 04 00:00:04.727881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.727845    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7311-4cb031"]
Nov 04 00:00:04.873838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.873798    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7311-179225"]
Nov 04 00:00:04.874851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.874828    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" containerName="test-curl-7311-4cb031"
Nov 04 00:00:04.874851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.874851    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" containerName="test-curl-7311-4cb031"
Nov 04 00:00:04.875097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.875084    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" containerName="test-curl-7311-4cb031"
Nov 04 00:00:04.883007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.882982    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:04.884499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.884470    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-179225"]
Nov 04 00:00:04.967534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.967489    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerStarted","Data":"265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860"}
Nov 04 00:00:04.970675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.970643    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerStarted","Data":"e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d"}
Nov 04 00:00:04.974109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.973789    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm" event={"ID":"5f2e92fa-4fbe-44b8-a3f6-b700da748344","Type":"ContainerStarted","Data":"5b068cf7ea1e68661ad54a9ff3125c2e976367e5dcb9e2ceb4494c406922d3b5"}
Nov 04 00:00:04.978033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.978003    5074 generic.go:356] "Generic (PLEG): container finished" podID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerID="3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd" exitCode=60
Nov 04 00:00:04.978159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.978102    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerDied","Data":"3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd"}
Nov 04 00:00:04.978269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.978164    5074 scope.go:122] "RemoveContainer" containerID="31a4df28e18f53030819d334c1d4f93bf2f82a0326c6cbcc6876687ee3205353"
Nov 04 00:00:04.978334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.978317    5074 scope.go:122] "RemoveContainer" containerID="3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd"
Nov 04 00:00:04.978649 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:04.978620    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m_odh-ai-gateway-infra(ffc0c6f8-14dd-404b-af9c-2651eaf5cf75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75"
Nov 04 00:00:04.980296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.980272    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerStarted","Data":"d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc"}
Nov 04 00:00:04.982236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.982207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerStarted","Data":"2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9"}
Nov 04 00:00:04.984042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.984020    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerStarted","Data":"9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34"}
Nov 04 00:00:04.985506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.985477    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69bp2\" (UniqueName: \"kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2\") pod \"test-curl-7311-179225\" (UID: \"d4103949-df91-41a2-b040-456c13a45b43\") " pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:04.986985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.986963    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerStarted","Data":"76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d"}
Nov 04 00:00:04.989226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.989204    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerStarted","Data":"557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba"}
Nov 04 00:00:04.991435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.991410    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerID="38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642" exitCode=60
Nov 04 00:00:04.991536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.991490    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerDied","Data":"38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642"}
Nov 04 00:00:04.991699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.991680    5074 scope.go:122] "RemoveContainer" containerID="38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642"
Nov 04 00:00:04.991979 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:04.991959    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q_odh-ai-gateway-infra(e4397e82-ad37-4ccc-81c8-fed890af8186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186"
Nov 04 00:00:04.993836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.993812    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerStarted","Data":"806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373"}
Nov 04 00:00:04.995151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.995087    5074 scope.go:122] "RemoveContainer" containerID="c83f237d3a4ea867058366e45d108d06aa80f067e5e4ba10ae812ba35b6fbf95"
Nov 04 00:00:04.996147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:04.996129    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerStarted","Data":"2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65"}
Nov 04 00:00:05.040177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.040144    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23deb7ce-d111-40ce-805a-fa5bf2095905" path="/var/lib/kubelet/pods/23deb7ce-d111-40ce-805a-fa5bf2095905/volumes"
Nov 04 00:00:05.041832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.041805    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7c28c4-4a93-43b5-8058-52f6382dac77" path="/var/lib/kubelet/pods/3e7c28c4-4a93-43b5-8058-52f6382dac77/volumes"
Nov 04 00:00:05.088947 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.087247    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bp2\" (UniqueName: \"kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2\") pod \"test-curl-7311-179225\" (UID: \"d4103949-df91-41a2-b040-456c13a45b43\") " pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:05.120394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.120327    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bp2\" (UniqueName: \"kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2\") pod \"test-curl-7311-179225\" (UID: \"d4103949-df91-41a2-b040-456c13a45b43\") " pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:05.216024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.215996    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:05.530061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.529041    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:05.537686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.537648    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7311-179225"]
Nov 04 00:00:05.542113 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:05.542079    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4103949_df91_41a2_b040_456c13a45b43.slice/crio-e51455859b4c4861d77f8ceab268a0477f04bcaf8e9ad14b4ae21a3d49eb1bde WatchSource:0}: Error finding container e51455859b4c4861d77f8ceab268a0477f04bcaf8e9ad14b4ae21a3d49eb1bde: Status 404 returned error can't find the container with id e51455859b4c4861d77f8ceab268a0477f04bcaf8e9ad14b4ae21a3d49eb1bde
Nov 04 00:00:05.594493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.594462    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr\" (UniqueName: \"kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr\") pod \"373f1058-e636-45d6-a53b-cdef5a009136\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") "
Nov 04 00:00:05.594653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.594568    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca\" (UniqueName: \"kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca\") pod \"373f1058-e636-45d6-a53b-cdef5a009136\" (UID: \"373f1058-e636-45d6-a53b-cdef5a009136\") "
Nov 04 00:00:05.594999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.594975    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca" pod "373f1058-e636-45d6-a53b-cdef5a009136" (UID: "373f1058-e636-45d6-a53b-cdef5a009136"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:05.595148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.595130    5074 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f1058-e636-45d6-a53b-cdef5a009136-serviceca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:05.601187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.601161    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr" pod "373f1058-e636-45d6-a53b-cdef5a009136" (UID: "373f1058-e636-45d6-a53b-cdef5a009136"). InnerVolumeSpecName "kube-api-access-v25kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:05.713761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:05.710590    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v25kr\" (UniqueName: \"kubernetes.io/projected/373f1058-e636-45d6-a53b-cdef5a009136-kube-api-access-v25kr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:06.008343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.008305    5074 generic.go:356] "Generic (PLEG): container finished" podID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerID="265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860" exitCode=60
Nov 04 00:00:06.008869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.008392    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerDied","Data":"265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860"}
Nov 04 00:00:06.008869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.008461    5074 scope.go:122] "RemoveContainer" containerID="0a79a26b3073ca9bea586358d7952f8210f32e37c86992ca7b358c555a36bae4"
Nov 04 00:00:06.008869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.008796    5074 scope.go:122] "RemoveContainer" containerID="265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860"
Nov 04 00:00:06.009255 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.009229    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7_odh-ai-gateway-infra(f6fe77b6-5988-47a8-88d2-103a2df70893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893"
Nov 04 00:00:06.010985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.010859    5074 generic.go:356] "Generic (PLEG): container finished" podID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerID="e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d" exitCode=60
Nov 04 00:00:06.010985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.010949    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerDied","Data":"e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d"}
Nov 04 00:00:06.011172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.011156    5074 scope.go:122] "RemoveContainer" containerID="e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d"
Nov 04 00:00:06.011437 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.011413    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb_odh-ai-gateway-infra(1743ec1d-9077-4c5c-85a1-2cfe5f7232c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6"
Nov 04 00:00:06.013991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.013969    5074 generic.go:356] "Generic (PLEG): container finished" podID="5f2e92fa-4fbe-44b8-a3f6-b700da748344" containerID="63397aca03160066fccdd59859470886382e857944fe8926d134b80c71129856" exitCode=0
Nov 04 00:00:06.014121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.014057    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm" event={"ID":"5f2e92fa-4fbe-44b8-a3f6-b700da748344","Type":"ContainerDied","Data":"63397aca03160066fccdd59859470886382e857944fe8926d134b80c71129856"}
Nov 04 00:00:06.016514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.016493    5074 scope.go:122] "RemoveContainer" containerID="3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd"
Nov 04 00:00:06.016823 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.016784    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m_odh-ai-gateway-infra(ffc0c6f8-14dd-404b-af9c-2651eaf5cf75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75"
Nov 04 00:00:06.019665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.019643    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3fab351-0452-48f0-9d68-260c672a0e85" containerID="d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc" exitCode=60
Nov 04 00:00:06.020091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.020071    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerDied","Data":"d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc"}
Nov 04 00:00:06.020699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.020669    5074 scope.go:122] "RemoveContainer" containerID="d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc"
Nov 04 00:00:06.021000 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.020972    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx_odh-ai-gateway-infra(a3fab351-0452-48f0-9d68-260c672a0e85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" podUID="a3fab351-0452-48f0-9d68-260c672a0e85"
Nov 04 00:00:06.022817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.022788    5074 generic.go:356] "Generic (PLEG): container finished" podID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerID="2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9" exitCode=60
Nov 04 00:00:06.022911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.022895    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerDied","Data":"2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9"}
Nov 04 00:00:06.023227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.023205    5074 scope.go:122] "RemoveContainer" containerID="2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9"
Nov 04 00:00:06.023514 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.023486    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9_odh-ai-gateway-infra(9555ad04-1dc8-4a33-b8d1-c17feae69865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865"
Nov 04 00:00:06.026164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.026140    5074 generic.go:356] "Generic (PLEG): container finished" podID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerID="9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34" exitCode=60
Nov 04 00:00:06.026257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.026217    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerDied","Data":"9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34"}
Nov 04 00:00:06.026424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.026403    5074 scope.go:122] "RemoveContainer" containerID="9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34"
Nov 04 00:00:06.026682 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.026635    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62_odh-ai-gateway-infra(22c21e35-deb3-4c06-ab24-9c372d4f1ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0"
Nov 04 00:00:06.028435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.028418    5074 scope.go:122] "RemoveContainer" containerID="8c3ad56731eeef47ae41f7f1e1283b7181df1005ce03950b357c388e0672d978"
Nov 04 00:00:06.028971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.028948    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-179225" event={"ID":"d4103949-df91-41a2-b040-456c13a45b43","Type":"ContainerStarted","Data":"f77c3428ad7f8df1390e570f9ff03666aa2089eded54e7728071090aa57bb921"}
Nov 04 00:00:06.029110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.029089    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7311-179225" event={"ID":"d4103949-df91-41a2-b040-456c13a45b43","Type":"ContainerStarted","Data":"e51455859b4c4861d77f8ceab268a0477f04bcaf8e9ad14b4ae21a3d49eb1bde"}
Nov 04 00:00:06.032062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.032038    5074 generic.go:356] "Generic (PLEG): container finished" podID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerID="76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d" exitCode=60
Nov 04 00:00:06.032157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.032123    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerDied","Data":"76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d"}
Nov 04 00:00:06.032347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.032324    5074 scope.go:122] "RemoveContainer" containerID="76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d"
Nov 04 00:00:06.032602 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.032578    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv_odh-ai-gateway-infra(c34218e8-bb4f-417b-9889-0fe3a0a5b486)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486"
Nov 04 00:00:06.033799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.033763    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29895840-b78h6" event={"ID":"373f1058-e636-45d6-a53b-cdef5a009136","Type":"ContainerDied","Data":"5d438a8fa7a59008d919d400a2474ef47a4b41067cc4ebc7d7d1b3b5f8bf9009"}
Nov 04 00:00:06.033884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.033806    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d438a8fa7a59008d919d400a2474ef47a4b41067cc4ebc7d7d1b3b5f8bf9009"
Nov 04 00:00:06.033884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.033816    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29895840-b78h6"
Nov 04 00:00:06.038120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.038097    5074 generic.go:356] "Generic (PLEG): container finished" podID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerID="557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba" exitCode=60
Nov 04 00:00:06.038217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.038179    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerDied","Data":"557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba"}
Nov 04 00:00:06.038429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.038406    5074 scope.go:122] "RemoveContainer" containerID="557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba"
Nov 04 00:00:06.038694 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.038669    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv_odh-ai-gateway-infra(0f9cf5b2-c042-4d3c-9419-6506fb8ddba9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9"
Nov 04 00:00:06.046807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.046652    5074 scope.go:122] "RemoveContainer" containerID="4bf2aa1883e386f91828d9ca4b93161620a7e168a9f3cf61dd941f3670f22d63"
Nov 04 00:00:06.047297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.047280    5074 scope.go:122] "RemoveContainer" containerID="38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642"
Nov 04 00:00:06.047634 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.047611    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q_odh-ai-gateway-infra(e4397e82-ad37-4ccc-81c8-fed890af8186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186"
Nov 04 00:00:06.049793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.049765    5074 generic.go:356] "Generic (PLEG): container finished" podID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerID="806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373" exitCode=60
Nov 04 00:00:06.050251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.050228    5074 scope.go:122] "RemoveContainer" containerID="806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373"
Nov 04 00:00:06.050596 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.050573    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr_odh-ai-gateway-infra(24761cdc-c09a-4aec-a99d-1b30c250a630)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630"
Nov 04 00:00:06.050647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.050621    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerDied","Data":"806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373"}
Nov 04 00:00:06.052892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.052870    5074 generic.go:356] "Generic (PLEG): container finished" podID="94ce3fbf-89e5-4766-a524-968f410af728" containerID="2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65" exitCode=60
Nov 04 00:00:06.052980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.052955    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerDied","Data":"2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65"}
Nov 04 00:00:06.053074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.053045    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7311-4cb031" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" containerName="test-curl-7311-4cb031" containerID="cri-o://cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949" gracePeriod=2
Nov 04 00:00:06.053456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.053261    5074 scope.go:122] "RemoveContainer" containerID="2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65"
Nov 04 00:00:06.053654 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.053629    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m_odh-ai-gateway-infra(94ce3fbf-89e5-4766-a524-968f410af728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" podUID="94ce3fbf-89e5-4766-a524-968f410af728"
Nov 04 00:00:06.063696 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.063677    5074 scope.go:122] "RemoveContainer" containerID="89746a56e838455de1f97c3adefdd0f18860fede007c4ff8cd535cc92ae5914f"
Nov 04 00:00:06.079966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.079938    5074 scope.go:122] "RemoveContainer" containerID="978fc6c4865c9291b267d5a374305256219329ee44e911ca4a96b33309f7a7aa"
Nov 04 00:00:06.094458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.094437    5074 scope.go:122] "RemoveContainer" containerID="adece17bfcb0d04f884d9c327f22a4c907e437b921b341ced42ef8e158092ffd"
Nov 04 00:00:06.110591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.110568    5074 scope.go:122] "RemoveContainer" containerID="65aa3399a1c4f7b8b8c6bfc021ed6968187eb736ee65d29bc3628b63c9d9cfef"
Nov 04 00:00:06.129202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.129176    5074 scope.go:122] "RemoveContainer" containerID="52b8073b7a60238728ed52b70dfe0736bcec4eebab1de64f7d5f1049e2038e04"
Nov 04 00:00:06.148281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.148260    5074 scope.go:122] "RemoveContainer" containerID="aa9ab6c127725723852fe515cea48b4951c48382019497510159355e45f23dff"
Nov 04 00:00:06.230109 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:06.230061    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-4cb031\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:06.268916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:06.268815    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7311-179225" podStartSLOduration=2.268801907 podStartE2EDuration="2.268801907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:06.264307104 +0000 UTC m=+1871.363907249" watchObservedRunningTime="2026-11-04 00:00:06.268801907 +0000 UTC m=+1871.368402051"
Nov 04 00:00:07.063288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.063254    5074 scope.go:122] "RemoveContainer" containerID="265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860"
Nov 04 00:00:07.063785 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.063653    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7_odh-ai-gateway-infra(f6fe77b6-5988-47a8-88d2-103a2df70893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893"
Nov 04 00:00:07.064999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.064974    5074 scope.go:122] "RemoveContainer" containerID="e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d"
Nov 04 00:00:07.065316 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.065289    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb_odh-ai-gateway-infra(1743ec1d-9077-4c5c-85a1-2cfe5f7232c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6"
Nov 04 00:00:07.067110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.067085    5074 scope.go:122] "RemoveContainer" containerID="d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc"
Nov 04 00:00:07.067621 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.067594    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx_odh-ai-gateway-infra(a3fab351-0452-48f0-9d68-260c672a0e85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" podUID="a3fab351-0452-48f0-9d68-260c672a0e85"
Nov 04 00:00:07.068711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.068694    5074 scope.go:122] "RemoveContainer" containerID="2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9"
Nov 04 00:00:07.069001 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.068979    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9_odh-ai-gateway-infra(9555ad04-1dc8-4a33-b8d1-c17feae69865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865"
Nov 04 00:00:07.070299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.070282    5074 scope.go:122] "RemoveContainer" containerID="9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34"
Nov 04 00:00:07.070564 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.070545    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62_odh-ai-gateway-infra(22c21e35-deb3-4c06-ab24-9c372d4f1ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0"
Nov 04 00:00:07.071937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.071921    5074 scope.go:122] "RemoveContainer" containerID="76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d"
Nov 04 00:00:07.072190 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.072167    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv_odh-ai-gateway-infra(c34218e8-bb4f-417b-9889-0fe3a0a5b486)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486"
Nov 04 00:00:07.073489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.073472    5074 scope.go:122] "RemoveContainer" containerID="557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba"
Nov 04 00:00:07.073707 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.073689    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv_odh-ai-gateway-infra(0f9cf5b2-c042-4d3c-9419-6506fb8ddba9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9"
Nov 04 00:00:07.075026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.075009    5074 scope.go:122] "RemoveContainer" containerID="806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373"
Nov 04 00:00:07.075268 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.075244    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr_odh-ai-gateway-infra(24761cdc-c09a-4aec-a99d-1b30c250a630)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630"
Nov 04 00:00:07.076670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.076652    5074 scope.go:122] "RemoveContainer" containerID="2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65"
Nov 04 00:00:07.077138 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:07.076879    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m_odh-ai-gateway-infra(94ce3fbf-89e5-4766-a524-968f410af728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" podUID="94ce3fbf-89e5-4766-a524-968f410af728"
Nov 04 00:00:07.289412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.289378    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:07.451332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.451288    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca\") pod \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") "
Nov 04 00:00:07.451514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.451370    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k\" (UniqueName: \"kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k\") pod \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") "
Nov 04 00:00:07.451637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.451622    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca\") pod \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\" (UID: \"5f2e92fa-4fbe-44b8-a3f6-b700da748344\") "
Nov 04 00:00:07.452056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.452026    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca" pod "5f2e92fa-4fbe-44b8-a3f6-b700da748344" (UID: "5f2e92fa-4fbe-44b8-a3f6-b700da748344"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:07.452202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.452187    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:07.454081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.454047    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca" pod "5f2e92fa-4fbe-44b8-a3f6-b700da748344" (UID: "5f2e92fa-4fbe-44b8-a3f6-b700da748344"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:07.454081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.454061    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k" pod "5f2e92fa-4fbe-44b8-a3f6-b700da748344" (UID: "5f2e92fa-4fbe-44b8-a3f6-b700da748344"). InnerVolumeSpecName "kube-api-access-r752k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:07.554030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.553992    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e92fa-4fbe-44b8-a3f6-b700da748344-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:07.554030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:07.554029    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r752k\" (UniqueName: \"kubernetes.io/projected/5f2e92fa-4fbe-44b8-a3f6-b700da748344-kube-api-access-r752k\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:08.086494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.086470    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"
Nov 04 00:00:08.086828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.086488    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm" event={"ID":"5f2e92fa-4fbe-44b8-a3f6-b700da748344","Type":"ContainerDied","Data":"5b068cf7ea1e68661ad54a9ff3125c2e976367e5dcb9e2ceb4494c406922d3b5"}
Nov 04 00:00:08.086828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.086529    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b068cf7ea1e68661ad54a9ff3125c2e976367e5dcb9e2ceb4494c406922d3b5"
Nov 04 00:00:08.276328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.276301    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:08.279730 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:08.279701    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-4cb031\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:08.365587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.365511    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx\" (UniqueName: \"kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx\") pod \"9c00755e-e4a6-4679-9e54-68f85af58467\" (UID: \"9c00755e-e4a6-4679-9e54-68f85af58467\") "
Nov 04 00:00:08.367686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.367656    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx" pod "9c00755e-e4a6-4679-9e54-68f85af58467" (UID: "9c00755e-e4a6-4679-9e54-68f85af58467"). InnerVolumeSpecName "kube-api-access-kqcbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:08.467609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:08.467576    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcbx\" (UniqueName: \"kubernetes.io/projected/9c00755e-e4a6-4679-9e54-68f85af58467-kube-api-access-kqcbx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:09.093763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.092782    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" path="/var/lib/kubelet/pods/9c00755e-e4a6-4679-9e54-68f85af58467/volumes"
Nov 04 00:00:09.094272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.093824    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"]
Nov 04 00:00:09.094272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.093899    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e83219f-4nh6l"]
Nov 04 00:00:09.111024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.110559    5074 generic.go:356] "Generic (PLEG): container finished" podID="9c00755e-e4a6-4679-9e54-68f85af58467" containerID="cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949" exitCode=137
Nov 04 00:00:09.111024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.110643    5074 scope.go:122] "RemoveContainer" containerID="cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949"
Nov 04 00:00:09.111024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.110827    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:09.121785 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:09.117991    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-4cb031\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:09.126170 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:09.126123    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-4cb031\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="9c00755e-e4a6-4679-9e54-68f85af58467" pod="opendatahub/test-curl-7311-4cb031"
Nov 04 00:00:09.130351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.130330    5074 scope.go:122] "RemoveContainer" containerID="cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949"
Nov 04 00:00:09.130727 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:09.130696    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949\": container with ID starting with cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949 not found: ID does not exist" containerID="cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949"
Nov 04 00:00:09.130829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:09.130790    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949"} err="failed to get container status \"cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949\": rpc error: code = NotFound desc = could not find container \"cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949\": container with ID starting with cbf277b4a271d0955148015db5aae72d2a19b3af794175cfe0d06f6b1fbac949 not found: ID does not exist"
Nov 04 00:00:10.384772 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:10.384721    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:11.022715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:11.022677    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22afadfe-91ed-4263-a27d-9e0b342496da" path="/var/lib/kubelet/pods/22afadfe-91ed-4263-a27d-9e0b342496da/volumes"
Nov 04 00:00:14.620453 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:14.620403    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:15.017272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:15.017235    5074 scope.go:122] "RemoveContainer" containerID="38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642"
Nov 04 00:00:15.694410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:15.694367    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"]
Nov 04 00:00:16.010565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:16.010472    5074 scope.go:122] "RemoveContainer" containerID="265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860"
Nov 04 00:00:16.010733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:16.010581    5074 scope.go:122] "RemoveContainer" containerID="557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba"
Nov 04 00:00:16.179458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:16.179422    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerStarted","Data":"896983129f03a198883b5d4938bc5bc916d39d9cdee69ff31d4ccfce904b2006"}
Nov 04 00:00:16.330619 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:16.329279    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:16.330619 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:16.329940    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:16.990320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:16.989865    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7311-179225"]
Nov 04 00:00:16.990320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:16.990133    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7311-179225" podUID="d4103949-df91-41a2-b040-456c13a45b43" containerName="test-curl-7311-179225" containerID="cri-o://f77c3428ad7f8df1390e570f9ff03666aa2089eded54e7728071090aa57bb921" gracePeriod=2
Nov 04 00:00:17.003229 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:17.003187    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:17.024768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.011671    5074 scope.go:122] "RemoveContainer" containerID="9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34"
Nov 04 00:00:17.024768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.012791    5074 scope.go:122] "RemoveContainer" containerID="3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd"
Nov 04 00:00:17.035425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.035356    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7311-179225"]
Nov 04 00:00:17.189476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.189446    5074 generic.go:356] "Generic (PLEG): container finished" podID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerID="d7d767d4c68fd5c1faab7426010bf3ff1ccd2e208d201f291efb4341fb35a0a6" exitCode=60
Nov 04 00:00:17.189653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.189518    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerDied","Data":"d7d767d4c68fd5c1faab7426010bf3ff1ccd2e208d201f291efb4341fb35a0a6"}
Nov 04 00:00:17.189653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.189575    5074 scope.go:122] "RemoveContainer" containerID="557d61d53fb3edb7cd9c7d679cab37583d1ed6c8f84079e0f1b6001da1ac94ba"
Nov 04 00:00:17.190055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.190037    5074 scope.go:122] "RemoveContainer" containerID="d7d767d4c68fd5c1faab7426010bf3ff1ccd2e208d201f291efb4341fb35a0a6"
Nov 04 00:00:17.190394 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:17.190376    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv_odh-ai-gateway-infra(0f9cf5b2-c042-4d3c-9419-6506fb8ddba9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9"
Nov 04 00:00:17.192446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.192424    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerID="896983129f03a198883b5d4938bc5bc916d39d9cdee69ff31d4ccfce904b2006" exitCode=60
Nov 04 00:00:17.192546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.192496    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerDied","Data":"896983129f03a198883b5d4938bc5bc916d39d9cdee69ff31d4ccfce904b2006"}
Nov 04 00:00:17.192921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.192900    5074 scope.go:122] "RemoveContainer" containerID="896983129f03a198883b5d4938bc5bc916d39d9cdee69ff31d4ccfce904b2006"
Nov 04 00:00:17.193305 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:17.193280    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q_odh-ai-gateway-infra(e4397e82-ad37-4ccc-81c8-fed890af8186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186"
Nov 04 00:00:17.194545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.194526    5074 generic.go:356] "Generic (PLEG): container finished" podID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerID="cefe0d560091b478962d339a5b255cab027968b1663716c21687b79799cb088e" exitCode=60
Nov 04 00:00:17.194637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.194593    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerDied","Data":"cefe0d560091b478962d339a5b255cab027968b1663716c21687b79799cb088e"}
Nov 04 00:00:17.194951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.194937    5074 scope.go:122] "RemoveContainer" containerID="cefe0d560091b478962d339a5b255cab027968b1663716c21687b79799cb088e"
Nov 04 00:00:17.195207 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:17.195190    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7_odh-ai-gateway-infra(f6fe77b6-5988-47a8-88d2-103a2df70893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893"
Nov 04 00:00:17.242670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.242641    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"]
Nov 04 00:00:17.247702 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:17.247662    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:17.280995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.280969    5074 scope.go:122] "RemoveContainer" containerID="38b7d2c70e8e341fffb325a2c26c773adba7c619432609bb94068aa02bf52642"
Nov 04 00:00:17.296666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:17.296643    5074 scope.go:122] "RemoveContainer" containerID="265d818e033f84fb6842dfec803516bde94e65e9fda9264cfac90fdb32643860"
Nov 04 00:00:17.308545 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:17.308474    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.010732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.010705    5074 scope.go:122] "RemoveContainer" containerID="2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9"
Nov 04 00:00:18.208117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.208083    5074 generic.go:356] "Generic (PLEG): container finished" podID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerID="d3bd95cae7679c26181488f0023e790f2976e8c8e1a73c957d2fb3800e7047b0" exitCode=60
Nov 04 00:00:18.208235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.208116    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerDied","Data":"d3bd95cae7679c26181488f0023e790f2976e8c8e1a73c957d2fb3800e7047b0"}
Nov 04 00:00:18.208235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.208196    5074 scope.go:122] "RemoveContainer" containerID="9fad42cfcc064cf3832943fc9c29f40d53449526252b778f8e9c3fbaaf274c34"
Nov 04 00:00:18.208677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.208656    5074 scope.go:122] "RemoveContainer" containerID="d3bd95cae7679c26181488f0023e790f2976e8c8e1a73c957d2fb3800e7047b0"
Nov 04 00:00:18.209126 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.209097    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62_odh-ai-gateway-infra(22c21e35-deb3-4c06-ab24-9c372d4f1ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0"
Nov 04 00:00:18.212109 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.212074    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.224956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.224929    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerStarted","Data":"09b45f9641532de7a4670e94d5379ae4510eb717566ef885faf85e41daccc0e9"}
Nov 04 00:00:18.255382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.255310    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"]
Nov 04 00:00:18.290256 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.290221    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.322152 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.322108    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.359583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.359492    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"]
Nov 04 00:00:18.367121 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.367080    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.559349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.559323    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:18.564705 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.563033    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.635027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.634999    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:18.676447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.676417    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca\") pod \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") "
Nov 04 00:00:18.676586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.676482    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx\" (UniqueName: \"kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx\") pod \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\" (UID: \"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9\") "
Nov 04 00:00:18.676586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.676550    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca\") pod \"e4397e82-ad37-4ccc-81c8-fed890af8186\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") "
Nov 04 00:00:18.676722 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.676685    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf\" (UniqueName: \"kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf\") pod \"e4397e82-ad37-4ccc-81c8-fed890af8186\" (UID: \"e4397e82-ad37-4ccc-81c8-fed890af8186\") "
Nov 04 00:00:18.679995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.679966    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf" pod "e4397e82-ad37-4ccc-81c8-fed890af8186" (UID: "e4397e82-ad37-4ccc-81c8-fed890af8186"). InnerVolumeSpecName "kube-api-access-dcmcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:18.680280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.680260    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca" pod "0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" (UID: "0f9cf5b2-c042-4d3c-9419-6506fb8ddba9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:18.682130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.682105    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx" pod "0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" (UID: "0f9cf5b2-c042-4d3c-9419-6506fb8ddba9"). InnerVolumeSpecName "kube-api-access-9xnrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:18.682410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.682392    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca" pod "e4397e82-ad37-4ccc-81c8-fed890af8186" (UID: "e4397e82-ad37-4ccc-81c8-fed890af8186"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:18.682968 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.682934    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.692966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.692936    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:18.744277 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:18.744238    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:18.784396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.782184    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz\" (UniqueName: \"kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz\") pod \"f6fe77b6-5988-47a8-88d2-103a2df70893\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") "
Nov 04 00:00:18.784396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.782723    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca\") pod \"f6fe77b6-5988-47a8-88d2-103a2df70893\" (UID: \"f6fe77b6-5988-47a8-88d2-103a2df70893\") "
Nov 04 00:00:18.784916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.784888    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:18.785008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.784923    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xnrx\" (UniqueName: \"kubernetes.io/projected/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9-kube-api-access-9xnrx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:18.785008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.784940    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4397e82-ad37-4ccc-81c8-fed890af8186-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:18.785008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.784954    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmcf\" (UniqueName: \"kubernetes.io/projected/e4397e82-ad37-4ccc-81c8-fed890af8186-kube-api-access-dcmcf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:18.785379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.785351    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca" pod "f6fe77b6-5988-47a8-88d2-103a2df70893" (UID: "f6fe77b6-5988-47a8-88d2-103a2df70893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:18.792294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.792262    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz" pod "f6fe77b6-5988-47a8-88d2-103a2df70893" (UID: "f6fe77b6-5988-47a8-88d2-103a2df70893"). InnerVolumeSpecName "kube-api-access-8l4fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:18.886960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.886919    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fe77b6-5988-47a8-88d2-103a2df70893-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:18.886960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:18.886950    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4fz\" (UniqueName: \"kubernetes.io/projected/f6fe77b6-5988-47a8-88d2-103a2df70893-kube-api-access-8l4fz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:19.010780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.010354    5074 scope.go:122] "RemoveContainer" containerID="806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373"
Nov 04 00:00:19.011532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.011340    5074 scope.go:122] "RemoveContainer" containerID="2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65"
Nov 04 00:00:19.012143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.012123    5074 scope.go:122] "RemoveContainer" containerID="76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d"
Nov 04 00:00:19.014040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.013509    5074 scope.go:122] "RemoveContainer" containerID="e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d"
Nov 04 00:00:19.017191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.017101    5074 scope.go:122] "RemoveContainer" containerID="d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc"
Nov 04 00:00:19.318058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.318027    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"]
Nov 04 00:00:19.558844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.558434    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"]
Nov 04 00:00:19.613372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.613292    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7" event={"ID":"f6fe77b6-5988-47a8-88d2-103a2df70893","Type":"ContainerDied","Data":"a9624979471b1b79031e70d822e0ca17983b855b5065c5d206acc691158a1f84"}
Nov 04 00:00:19.613564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.613547    5074 scope.go:122] "RemoveContainer" containerID="cefe0d560091b478962d339a5b255cab027968b1663716c21687b79799cb088e"
Nov 04 00:00:19.613972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.613949    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"
Nov 04 00:00:19.623289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.623266    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:19.624527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.624496    5074 generic.go:356] "Generic (PLEG): container finished" podID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerID="09b45f9641532de7a4670e94d5379ae4510eb717566ef885faf85e41daccc0e9" exitCode=60
Nov 04 00:00:19.624674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.624655    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerDied","Data":"09b45f9641532de7a4670e94d5379ae4510eb717566ef885faf85e41daccc0e9"}
Nov 04 00:00:19.633036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.633011    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerStarted","Data":"1a54b1413562f5ff90f4fbdaaf4b36848b9b93622d0ba740d07f50e2dae10445"}
Nov 04 00:00:19.633497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.633480    5074 scope.go:122] "RemoveContainer" containerID="1a54b1413562f5ff90f4fbdaaf4b36848b9b93622d0ba740d07f50e2dae10445"
Nov 04 00:00:19.634476 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:19.634436    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9_odh-ai-gateway-infra(9555ad04-1dc8-4a33-b8d1-c17feae69865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865"
Nov 04 00:00:19.646045 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.646019    5074 generic.go:356] "Generic (PLEG): container finished" podID="d4103949-df91-41a2-b040-456c13a45b43" containerID="f77c3428ad7f8df1390e570f9ff03666aa2089eded54e7728071090aa57bb921" exitCode=137
Nov 04 00:00:19.649683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.648187    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:19.665791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.665710    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv" event={"ID":"0f9cf5b2-c042-4d3c-9419-6506fb8ddba9","Type":"ContainerDied","Data":"d8adcab579157e3f6c7121dbe9979dd1692639501cdd03ffbb3d2ea83db546ab"}
Nov 04 00:00:19.669507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.668549    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"
Nov 04 00:00:19.674092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.672979    5074 scope.go:122] "RemoveContainer" containerID="3081b296e61b560e5c7af685ab2bed9f56b858cc70e0c5c4d6bd05a0e231d6fd"
Nov 04 00:00:19.674092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.673875    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q" event={"ID":"e4397e82-ad37-4ccc-81c8-fed890af8186","Type":"ContainerDied","Data":"991dc4c008b91353047f5b83db129dae926ec00f37b2e020e19c3d8c19fb736a"}
Nov 04 00:00:19.674283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.674091    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"
Nov 04 00:00:19.693938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.692918    5074 scope.go:122] "RemoveContainer" containerID="f77c3428ad7f8df1390e570f9ff03666aa2089eded54e7728071090aa57bb921"
Nov 04 00:00:19.733258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.733224    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2\" (UniqueName: \"kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2\") pod \"d4103949-df91-41a2-b040-456c13a45b43\" (UID: \"d4103949-df91-41a2-b040-456c13a45b43\") "
Nov 04 00:00:19.744413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.744088    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2" pod "d4103949-df91-41a2-b040-456c13a45b43" (UID: "d4103949-df91-41a2-b040-456c13a45b43"). InnerVolumeSpecName "kube-api-access-69bp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:19.744413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.743808    5074 scope.go:122] "RemoveContainer" containerID="d7d767d4c68fd5c1faab7426010bf3ff1ccd2e208d201f291efb4341fb35a0a6"
Nov 04 00:00:19.768891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.768112    5074 scope.go:122] "RemoveContainer" containerID="896983129f03a198883b5d4938bc5bc916d39d9cdee69ff31d4ccfce904b2006"
Nov 04 00:00:19.795368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.795328    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"]
Nov 04 00:00:19.806446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.806409    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895840-8ddv7"]
Nov 04 00:00:19.834694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.834669    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"]
Nov 04 00:00:19.841387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.841357    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895840-s4k7q"]
Nov 04 00:00:19.841850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.841751    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69bp2\" (UniqueName: \"kubernetes.io/projected/d4103949-df91-41a2-b040-456c13a45b43-kube-api-access-69bp2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:19.859786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.859731    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"]
Nov 04 00:00:19.878988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.878957    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895840-mz4rv"]
Nov 04 00:00:19.983010 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:19.982956    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:19.987625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.987601    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:19.990933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:19.990915    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:20.013608 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.013577    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:20.042530 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.042489    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:20.165822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.164719    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf\" (UniqueName: \"kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf\") pod \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") "
Nov 04 00:00:20.165822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.164926    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x\" (UniqueName: \"kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x\") pod \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") "
Nov 04 00:00:20.165822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.165001    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca\") pod \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\" (UID: \"22c21e35-deb3-4c06-ab24-9c372d4f1ba0\") "
Nov 04 00:00:20.165822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.165195    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca\") pod \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\" (UID: \"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75\") "
Nov 04 00:00:20.165822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.165772    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca" pod "ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" (UID: "ffc0c6f8-14dd-404b-af9c-2651eaf5cf75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:20.166233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.166146    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca" pod "22c21e35-deb3-4c06-ab24-9c372d4f1ba0" (UID: "22c21e35-deb3-4c06-ab24-9c372d4f1ba0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:20.167840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.167790    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x" pod "ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" (UID: "ffc0c6f8-14dd-404b-af9c-2651eaf5cf75"). InnerVolumeSpecName "kube-api-access-mhh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:20.168178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.168157    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf" pod "22c21e35-deb3-4c06-ab24-9c372d4f1ba0" (UID: "22c21e35-deb3-4c06-ab24-9c372d4f1ba0"). InnerVolumeSpecName "kube-api-access-5kqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:20.267923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.267887    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhh9x\" (UniqueName: \"kubernetes.io/projected/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-kube-api-access-mhh9x\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:20.267923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.267919    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:20.267923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.267932    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:20.268235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.267941    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kqdf\" (UniqueName: \"kubernetes.io/projected/22c21e35-deb3-4c06-ab24-9c372d4f1ba0-kube-api-access-5kqdf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:20.406335 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.406287    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:20.684164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.684127    5074 generic.go:356] "Generic (PLEG): container finished" podID="94ce3fbf-89e5-4766-a524-968f410af728" containerID="b63b00097dabf18c1064a47cd4313af9e0908c46664d287781aeccf72accb92c" exitCode=60
Nov 04 00:00:20.684354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.684195    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerDied","Data":"b63b00097dabf18c1064a47cd4313af9e0908c46664d287781aeccf72accb92c"}
Nov 04 00:00:20.684354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.684264    5074 scope.go:122] "RemoveContainer" containerID="2712a008c75cd484d60aa40ac586e1812af4b2b3b9f96e9bbe47c74b96a06e65"
Nov 04 00:00:20.684724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.684702    5074 scope.go:122] "RemoveContainer" containerID="b63b00097dabf18c1064a47cd4313af9e0908c46664d287781aeccf72accb92c"
Nov 04 00:00:20.685089 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.685066    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m_odh-ai-gateway-infra(94ce3fbf-89e5-4766-a524-968f410af728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" podUID="94ce3fbf-89e5-4766-a524-968f410af728"
Nov 04 00:00:20.688168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.688139    5074 generic.go:356] "Generic (PLEG): container finished" podID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerID="3eac0f10db67c5f7a9e9cadf96ae53969bfc4311171ace8099098277a0a7538b" exitCode=60
Nov 04 00:00:20.688311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.688292    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerDied","Data":"3eac0f10db67c5f7a9e9cadf96ae53969bfc4311171ace8099098277a0a7538b"}
Nov 04 00:00:20.688750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.688722    5074 scope.go:122] "RemoveContainer" containerID="3eac0f10db67c5f7a9e9cadf96ae53969bfc4311171ace8099098277a0a7538b"
Nov 04 00:00:20.689108 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.689082    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb_odh-ai-gateway-infra(1743ec1d-9077-4c5c-85a1-2cfe5f7232c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6"
Nov 04 00:00:20.689291 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.689268    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:20.690697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.690675    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"
Nov 04 00:00:20.690803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.690690    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m" event={"ID":"ffc0c6f8-14dd-404b-af9c-2651eaf5cf75","Type":"ContainerDied","Data":"317e96a48971bc6518f9d3de215598d29ef6c4588556166eca0191d247bed792"}
Nov 04 00:00:20.694892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.694868    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3fab351-0452-48f0-9d68-260c672a0e85" containerID="eb44fbc875b52fa613fe16c07a868dbcfe6020ca2725268e0429b2bd7ce46cf5" exitCode=60
Nov 04 00:00:20.694978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.694936    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerDied","Data":"eb44fbc875b52fa613fe16c07a868dbcfe6020ca2725268e0429b2bd7ce46cf5"}
Nov 04 00:00:20.695344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.695321    5074 scope.go:122] "RemoveContainer" containerID="eb44fbc875b52fa613fe16c07a868dbcfe6020ca2725268e0429b2bd7ce46cf5"
Nov 04 00:00:20.695637 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.695605    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx_odh-ai-gateway-infra(a3fab351-0452-48f0-9d68-260c672a0e85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" podUID="a3fab351-0452-48f0-9d68-260c672a0e85"
Nov 04 00:00:20.697161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.697141    5074 generic.go:356] "Generic (PLEG): container finished" podID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerID="1a54b1413562f5ff90f4fbdaaf4b36848b9b93622d0ba740d07f50e2dae10445" exitCode=60
Nov 04 00:00:20.697256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.697212    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerDied","Data":"1a54b1413562f5ff90f4fbdaaf4b36848b9b93622d0ba740d07f50e2dae10445"}
Nov 04 00:00:20.697567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.697548    5074 scope.go:122] "RemoveContainer" containerID="1a54b1413562f5ff90f4fbdaaf4b36848b9b93622d0ba740d07f50e2dae10445"
Nov 04 00:00:20.697825 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.697805    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9_odh-ai-gateway-infra(9555ad04-1dc8-4a33-b8d1-c17feae69865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865"
Nov 04 00:00:20.698631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.698615    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"
Nov 04 00:00:20.698716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.698630    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62" event={"ID":"22c21e35-deb3-4c06-ab24-9c372d4f1ba0","Type":"ContainerDied","Data":"125381af6f64ce45dd15be1b0a1ce61a8e19a2e8bd3d7599ce6df492e3b2a77e"}
Nov 04 00:00:20.701517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.701488    5074 scope.go:122] "RemoveContainer" containerID="e738cba07fab8fc8ee3d6f55c7f5f6edf96483271dd28485a168986b21296b7d"
Nov 04 00:00:20.702161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.702141    5074 generic.go:356] "Generic (PLEG): container finished" podID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerID="15a84bdbd88bf4950308e455b1c41d4852b99b1444ac5ce695102d6e73e74ec3" exitCode=60
Nov 04 00:00:20.702246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.702208    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerDied","Data":"15a84bdbd88bf4950308e455b1c41d4852b99b1444ac5ce695102d6e73e74ec3"}
Nov 04 00:00:20.702561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.702538    5074 scope.go:122] "RemoveContainer" containerID="15a84bdbd88bf4950308e455b1c41d4852b99b1444ac5ce695102d6e73e74ec3"
Nov 04 00:00:20.702913 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.702842    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv_odh-ai-gateway-infra(c34218e8-bb4f-417b-9889-0fe3a0a5b486)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486"
Nov 04 00:00:20.707621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.707597    5074 generic.go:356] "Generic (PLEG): container finished" podID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerID="abda8731d1a3ef48aff7f88ed339f43d51a4b5945247a49b7da57f2f775f8896" exitCode=60
Nov 04 00:00:20.707727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.707637    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerDied","Data":"abda8731d1a3ef48aff7f88ed339f43d51a4b5945247a49b7da57f2f775f8896"}
Nov 04 00:00:20.708056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.708033    5074 scope.go:122] "RemoveContainer" containerID="abda8731d1a3ef48aff7f88ed339f43d51a4b5945247a49b7da57f2f775f8896"
Nov 04 00:00:20.708330 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.708304    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr_odh-ai-gateway-infra(24761cdc-c09a-4aec-a99d-1b30c250a630)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630"
Nov 04 00:00:20.750999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.750968    5074 scope.go:122] "RemoveContainer" containerID="09b45f9641532de7a4670e94d5379ae4510eb717566ef885faf85e41daccc0e9"
Nov 04 00:00:20.771895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.771856    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"]
Nov 04 00:00:20.792201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.792172    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"]
Nov 04 00:00:20.815221 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.813377    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895840-8vk7m"]
Nov 04 00:00:20.815221 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.814590    5074 scope.go:122] "RemoveContainer" containerID="d5949ca6ea114c4ef7e6a61486233a1abf02524f60f875bb231bd17c02d010bc"
Nov 04 00:00:20.835629 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.835593    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:20.843407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.843294    5074 scope.go:122] "RemoveContainer" containerID="2fd35bed5443228621ff40adf80bf907bdc1c1a6f201ff110f3a495879b64ce9"
Nov 04 00:00:20.860952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.860931    5074 scope.go:122] "RemoveContainer" containerID="d3bd95cae7679c26181488f0023e790f2976e8c8e1a73c957d2fb3800e7047b0"
Nov 04 00:00:20.872647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.872622    5074 scope.go:122] "RemoveContainer" containerID="76941e652197233a8cd66f8d50b35174b0e3f83d1d79a80bfaba9a63a1b4bb8d"
Nov 04 00:00:20.889447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.889408    5074 scope.go:122] "RemoveContainer" containerID="806e2efc2fe5b71ef50ea2f665b11b1d0e4528958e53e340a2485fab2c8fd373"
Nov 04 00:00:20.915842 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:20.915807    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7311-179225\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="d4103949-df91-41a2-b040-456c13a45b43" pod="opendatahub/test-curl-7311-179225"
Nov 04 00:00:20.961520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.961431    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"]
Nov 04 00:00:20.971720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:20.971685    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895840-6tr62"]
Nov 04 00:00:21.020203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.020163    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" path="/var/lib/kubelet/pods/0f9cf5b2-c042-4d3c-9419-6506fb8ddba9/volumes"
Nov 04 00:00:21.020682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.020651    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" path="/var/lib/kubelet/pods/22c21e35-deb3-4c06-ab24-9c372d4f1ba0/volumes"
Nov 04 00:00:21.021141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.021121    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4103949-df91-41a2-b040-456c13a45b43" path="/var/lib/kubelet/pods/d4103949-df91-41a2-b040-456c13a45b43/volumes"
Nov 04 00:00:21.021830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.021808    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" path="/var/lib/kubelet/pods/e4397e82-ad37-4ccc-81c8-fed890af8186/volumes"
Nov 04 00:00:21.022249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.022233    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" path="/var/lib/kubelet/pods/f6fe77b6-5988-47a8-88d2-103a2df70893/volumes"
Nov 04 00:00:21.022629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.022615    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" path="/var/lib/kubelet/pods/ffc0c6f8-14dd-404b-af9c-2651eaf5cf75/volumes"
Nov 04 00:00:21.724389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.724359    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"]
Nov 04 00:00:21.774695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.774659    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"]
Nov 04 00:00:21.833249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.833209    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"]
Nov 04 00:00:21.851047 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.851012    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"]
Nov 04 00:00:21.910474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:21.910428    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"]
Nov 04 00:00:22.021117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.021089    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:22.131921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.131887    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca\") pod \"9555ad04-1dc8-4a33-b8d1-c17feae69865\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") "
Nov 04 00:00:22.132698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.132671    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc\" (UniqueName: \"kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc\") pod \"9555ad04-1dc8-4a33-b8d1-c17feae69865\" (UID: \"9555ad04-1dc8-4a33-b8d1-c17feae69865\") "
Nov 04 00:00:22.133454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.132499    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca" pod "9555ad04-1dc8-4a33-b8d1-c17feae69865" (UID: "9555ad04-1dc8-4a33-b8d1-c17feae69865"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:22.136439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.136419    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9555ad04-1dc8-4a33-b8d1-c17feae69865-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.144155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.143841    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:22.154422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.154367    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc" pod "9555ad04-1dc8-4a33-b8d1-c17feae69865" (UID: "9555ad04-1dc8-4a33-b8d1-c17feae69865"). InnerVolumeSpecName "kube-api-access-7n4rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.237398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.237362    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca\") pod \"94ce3fbf-89e5-4766-a524-968f410af728\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") "
Nov 04 00:00:22.237532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.237511    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2\" (UniqueName: \"kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2\") pod \"94ce3fbf-89e5-4766-a524-968f410af728\" (UID: \"94ce3fbf-89e5-4766-a524-968f410af728\") "
Nov 04 00:00:22.238815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.238786    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4rc\" (UniqueName: \"kubernetes.io/projected/9555ad04-1dc8-4a33-b8d1-c17feae69865-kube-api-access-7n4rc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.239977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.239946    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca" pod "94ce3fbf-89e5-4766-a524-968f410af728" (UID: "94ce3fbf-89e5-4766-a524-968f410af728"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:22.261924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.261879    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2" pod "94ce3fbf-89e5-4766-a524-968f410af728" (UID: "94ce3fbf-89e5-4766-a524-968f410af728"). InnerVolumeSpecName "kube-api-access-db7z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.262140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.262055    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:22.268080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.267289    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:22.274617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.274592    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:22.348271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.348226    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3fbf-89e5-4766-a524-968f410af728-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.351273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.351228    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db7z2\" (UniqueName: \"kubernetes.io/projected/94ce3fbf-89e5-4766-a524-968f410af728-kube-api-access-db7z2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.403047 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.403012    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:22.412055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.412007    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"]
Nov 04 00:00:22.415359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.415328    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:22.415606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.415584    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:22.415766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.415733    5074 cpu_manager.go:414] "containerMap: removing container" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:22.415874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.415860    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:22.415977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.415963    5074 cpu_manager.go:414] "containerMap: removing container" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.416041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.415977    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.416041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416002    5074 cpu_manager.go:414] "containerMap: removing container" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.416041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416010    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.416041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416029    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.416041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416037    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416053    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416062    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416073    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416080    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416093    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416100    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416113    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416120    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416131    5074 cpu_manager.go:414] "containerMap: removing container" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416138    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416152    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416159    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416167    5074 cpu_manager.go:414] "containerMap: removing container" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416174    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416189    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416197    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416207    5074 cpu_manager.go:414] "containerMap: removing container" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416214    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416227    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416234    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416247    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416254    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416265    5074 cpu_manager.go:414] "containerMap: removing container" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416272    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416281    5074 cpu_manager.go:414] "containerMap: removing container" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416288    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416304    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.416302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416310    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416638    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d4103949-df91-41a2-b040-456c13a45b43" containerName="test-curl-7311-179225"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416654    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4103949-df91-41a2-b040-456c13a45b43" containerName="test-curl-7311-179225"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416667    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416675    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416710    5074 cpu_manager.go:414] "containerMap: removing container" podUID="373f1058-e636-45d6-a53b-cdef5a009136" containerName="image-pruner"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416719    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373f1058-e636-45d6-a53b-cdef5a009136" containerName="image-pruner"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416780    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416788    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416805    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416813    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416830    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416837    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416858    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416865    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416876    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5f2e92fa-4fbe-44b8-a3f6-b700da748344" containerName="revoke-keys"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416883    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2e92fa-4fbe-44b8-a3f6-b700da748344" containerName="revoke-keys"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416897    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416904    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416918    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416925    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416937    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.416944    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417140    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417153    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417164    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417173    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417182    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2e92fa-4fbe-44b8-a3f6-b700da748344" containerName="revoke-keys"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417194    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417204    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417214    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417224    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417233    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417244    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417255    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417264    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417272    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417280    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417288    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417298    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417308    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417317    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417328    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417339    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417349    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ce3fbf-89e5-4766-a524-968f410af728" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417358    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417368    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4103949-df91-41a2-b040-456c13a45b43" containerName="test-curl-7311-179225"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417376    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417387    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="373f1058-e636-45d6-a53b-cdef5a009136" containerName="image-pruner"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417398    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417407    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:22.417683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.417419    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:22.422129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.422105    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.429883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.429579    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:00:22.443376 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.443349    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"]
Nov 04 00:00:22.465225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.465167    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca\") pod \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") "
Nov 04 00:00:22.465480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.465459    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca\") pod \"24761cdc-c09a-4aec-a99d-1b30c250a630\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") "
Nov 04 00:00:22.465645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.465626    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds\" (UniqueName: \"kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds\") pod \"24761cdc-c09a-4aec-a99d-1b30c250a630\" (UID: \"24761cdc-c09a-4aec-a99d-1b30c250a630\") "
Nov 04 00:00:22.465915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.465891    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2\" (UniqueName: \"kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2\") pod \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\" (UID: \"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6\") "
Nov 04 00:00:22.466077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.466060    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca\") pod \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") "
Nov 04 00:00:22.466256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.466222    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca\") pod \"a3fab351-0452-48f0-9d68-260c672a0e85\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") "
Nov 04 00:00:22.466440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.466403    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8\" (UniqueName: \"kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8\") pod \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\" (UID: \"c34218e8-bb4f-417b-9889-0fe3a0a5b486\") "
Nov 04 00:00:22.466580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.466561    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2\" (UniqueName: \"kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2\") pod \"a3fab351-0452-48f0-9d68-260c672a0e85\" (UID: \"a3fab351-0452-48f0-9d68-260c672a0e85\") "
Nov 04 00:00:22.467021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.466991    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.467282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.467264    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqpf\" (UniqueName: \"kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.467571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.467553    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.468244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.468221    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca" pod "1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" (UID: "1743ec1d-9077-4c5c-85a1-2cfe5f7232c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:22.468674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.468651    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca" pod "24761cdc-c09a-4aec-a99d-1b30c250a630" (UID: "24761cdc-c09a-4aec-a99d-1b30c250a630"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:22.472999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.472972    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds" pod "24761cdc-c09a-4aec-a99d-1b30c250a630" (UID: "24761cdc-c09a-4aec-a99d-1b30c250a630"). InnerVolumeSpecName "kube-api-access-m97ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.483470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.480009    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca" pod "a3fab351-0452-48f0-9d68-260c672a0e85" (UID: "a3fab351-0452-48f0-9d68-260c672a0e85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:22.484274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.483958    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca" pod "c34218e8-bb4f-417b-9889-0fe3a0a5b486" (UID: "c34218e8-bb4f-417b-9889-0fe3a0a5b486"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:22.487443 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.487418    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2" pod "1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" (UID: "1743ec1d-9077-4c5c-85a1-2cfe5f7232c6"). InnerVolumeSpecName "kube-api-access-7lkg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.507142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.507093    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8" pod "c34218e8-bb4f-417b-9889-0fe3a0a5b486" (UID: "c34218e8-bb4f-417b-9889-0fe3a0a5b486"). InnerVolumeSpecName "kube-api-access-bmxr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.509702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.509674    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2" pod "a3fab351-0452-48f0-9d68-260c672a0e85" (UID: "a3fab351-0452-48f0-9d68-260c672a0e85"). InnerVolumeSpecName "kube-api-access-ptnb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.569719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.569671    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqpf\" (UniqueName: \"kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.569938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.569919    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.570084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570068    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.570235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570221    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxr8\" (UniqueName: \"kubernetes.io/projected/c34218e8-bb4f-417b-9889-0fe3a0a5b486-kube-api-access-bmxr8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570241    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnb2\" (UniqueName: \"kubernetes.io/projected/a3fab351-0452-48f0-9d68-260c672a0e85-kube-api-access-ptnb2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570256    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570271    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24761cdc-c09a-4aec-a99d-1b30c250a630-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570286    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97ds\" (UniqueName: \"kubernetes.io/projected/24761cdc-c09a-4aec-a99d-1b30c250a630-kube-api-access-m97ds\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570302    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lkg2\" (UniqueName: \"kubernetes.io/projected/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6-kube-api-access-7lkg2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570317    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34218e8-bb4f-417b-9889-0fe3a0a5b486-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570332    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fab351-0452-48f0-9d68-260c672a0e85-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.570691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.570668    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.572591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.572568    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.583266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.583243    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqpf\" (UniqueName: \"kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf\") pod \"maas-api-revoke-sub-ef8ff953-rd5rs\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.758114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.758025    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"
Nov 04 00:00:22.758114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.758063    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb" event={"ID":"1743ec1d-9077-4c5c-85a1-2cfe5f7232c6","Type":"ContainerDied","Data":"9002f7066c93f19a91ac97316c5b25a6263609e1785f907dbd933fafded11a5f"}
Nov 04 00:00:22.758340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.758136    5074 scope.go:122] "RemoveContainer" containerID="3eac0f10db67c5f7a9e9cadf96ae53969bfc4311171ace8099098277a0a7538b"
Nov 04 00:00:22.759697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.759682    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"
Nov 04 00:00:22.759810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.759783    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx" event={"ID":"a3fab351-0452-48f0-9d68-260c672a0e85","Type":"ContainerDied","Data":"0c24e90cb582c33226af675ded2aecbb003448d465e0446bc2634d5b9da830d3"}
Nov 04 00:00:22.761559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.761523    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"
Nov 04 00:00:22.761679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.761602    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9" event={"ID":"9555ad04-1dc8-4a33-b8d1-c17feae69865","Type":"ContainerDied","Data":"13b64585f75fef5ec1ccacde52c3fc85137a7eabd777be25ee0321af1ea4be23"}
Nov 04 00:00:22.763677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.763657    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"
Nov 04 00:00:22.763807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.763724    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv" event={"ID":"c34218e8-bb4f-417b-9889-0fe3a0a5b486","Type":"ContainerDied","Data":"d10ca78c6bd48b8b3b3d204b6b36723419e9707e68cbda0628a9c9f35b8a325d"}
Nov 04 00:00:22.765784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.765761    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"
Nov 04 00:00:22.766063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.765763    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr" event={"ID":"24761cdc-c09a-4aec-a99d-1b30c250a630","Type":"ContainerDied","Data":"7b4ed2c2c069ddc4c2acc76aab6000f5688b892cd88ce102d8ea24d4bf3a0535"}
Nov 04 00:00:22.767429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.767406    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m" event={"ID":"94ce3fbf-89e5-4766-a524-968f410af728","Type":"ContainerDied","Data":"10b4fb106e9d6a9c62b1c2bb36fd1e35b0fd6b3878f8479e68106e2a32a3b8a7"}
Nov 04 00:00:22.767542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.767489    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"
Nov 04 00:00:22.773113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.773009    5074 scope.go:122] "RemoveContainer" containerID="eb44fbc875b52fa613fe16c07a868dbcfe6020ca2725268e0429b2bd7ce46cf5"
Nov 04 00:00:22.784304 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.784283    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:22.787807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.787732    5074 scope.go:122] "RemoveContainer" containerID="1a54b1413562f5ff90f4fbdaaf4b36848b9b93622d0ba740d07f50e2dae10445"
Nov 04 00:00:22.801607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.801583    5074 scope.go:122] "RemoveContainer" containerID="15a84bdbd88bf4950308e455b1c41d4852b99b1444ac5ce695102d6e73e74ec3"
Nov 04 00:00:22.813601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.813577    5074 scope.go:122] "RemoveContainer" containerID="abda8731d1a3ef48aff7f88ed339f43d51a4b5945247a49b7da57f2f775f8896"
Nov 04 00:00:22.829622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.829594    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"]
Nov 04 00:00:22.830358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.830328    5074 scope.go:122] "RemoveContainer" containerID="b63b00097dabf18c1064a47cd4313af9e0908c46664d287781aeccf72accb92c"
Nov 04 00:00:22.840165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.840143    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895840-24v9m"]
Nov 04 00:00:22.895297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.895264    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"]
Nov 04 00:00:22.914187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.914152    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895840-fn7rr"]
Nov 04 00:00:22.965691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.965654    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"]
Nov 04 00:00:22.978666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:22.978634    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895840-2tjzb"]
Nov 04 00:00:23.004259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.004227    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"]
Nov 04 00:00:23.031081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.030967    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1743ec1d-9077-4c5c-85a1-2cfe5f7232c6" path="/var/lib/kubelet/pods/1743ec1d-9077-4c5c-85a1-2cfe5f7232c6/volumes"
Nov 04 00:00:23.031971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.031941    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" path="/var/lib/kubelet/pods/24761cdc-c09a-4aec-a99d-1b30c250a630/volumes"
Nov 04 00:00:23.032707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.032683    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ce3fbf-89e5-4766-a524-968f410af728" path="/var/lib/kubelet/pods/94ce3fbf-89e5-4766-a524-968f410af728/volumes"
Nov 04 00:00:23.033634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.033611    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895840-5l2cx"]
Nov 04 00:00:23.096849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.096808    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"]
Nov 04 00:00:23.108555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.108522    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895840-z8vkv"]
Nov 04 00:00:23.152875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.152843    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"]
Nov 04 00:00:23.161790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.156765    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"]
Nov 04 00:00:23.161790 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:23.160518    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e13b562_d3c7_484f_a114_d33a6f5cd895.slice/crio-4f645e36b0045e88b10284d764dae80939dc293bf1e135afda0ad0334db61079 WatchSource:0}: Error finding container 4f645e36b0045e88b10284d764dae80939dc293bf1e135afda0ad0334db61079: Status 404 returned error can't find the container with id 4f645e36b0045e88b10284d764dae80939dc293bf1e135afda0ad0334db61079
Nov 04 00:00:23.168344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.166442    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895840-k7rq9"]
Nov 04 00:00:23.781866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.781825    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs" event={"ID":"4e13b562-d3c7-484f-a114-d33a6f5cd895","Type":"ContainerStarted","Data":"b315bed25e05991485f7a66d819f2cd651c99050ee8607b67c7d23f3156ebfd5"}
Nov 04 00:00:23.782080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.781885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs" event={"ID":"4e13b562-d3c7-484f-a114-d33a6f5cd895","Type":"ContainerStarted","Data":"4f645e36b0045e88b10284d764dae80939dc293bf1e135afda0ad0334db61079"}
Nov 04 00:00:23.821020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:23.820962    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs" podStartSLOduration=1.820947562 podStartE2EDuration="1.820947562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:23.817852303 +0000 UTC m=+1888.917452447" watchObservedRunningTime="2026-11-04 00:00:23.820947562 +0000 UTC m=+1888.920547707"
Nov 04 00:00:24.270603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.270560    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"]
Nov 04 00:00:24.271351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271329    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:24.271351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271351    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271378    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271384    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271392    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271397    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271406    5074 cpu_manager.go:414] "containerMap: removing container" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271412    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271428    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271432    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271447    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:24.271526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271452    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9cf5b2-c042-4d3c-9419-6506fb8ddba9" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271546    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24761cdc-c09a-4aec-a99d-1b30c250a630" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271558    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c21e35-deb3-4c06-ab24-9c372d4f1ba0" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271568    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271577    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fe77b6-5988-47a8-88d2-103a2df70893" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271584    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc0c6f8-14dd-404b-af9c-2651eaf5cf75" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271592    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4397e82-ad37-4ccc-81c8-fed890af8186" containerName="cleanup"
Nov 04 00:00:24.272034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.271599    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" containerName="cleanup"
Nov 04 00:00:24.275552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.275529    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.286996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.286967    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"]
Nov 04 00:00:24.290722 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.288852    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fa0d0a\""
Nov 04 00:00:24.321861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.321818    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"]
Nov 04 00:00:24.329721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.329680    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dvz\" (UniqueName: \"kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz\") pod \"payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.329880 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.329758    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.330184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.330163    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.357189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.357106    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fa0d0a-dockercfg-gl49z\""
Nov 04 00:00:24.361060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.360986    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"]
Nov 04 00:00:24.498984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.498943    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.499262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.499242    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbrm\" (UniqueName: \"kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm\") pod \"payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.499822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.499789    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dvz\" (UniqueName: \"kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz\") pod \"payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.513418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.513371    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.515490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.515447    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.556566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.556486    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dvz\" (UniqueName: \"kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz\") pod \"payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.606772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.606526    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:24.615609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.615547    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.615913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.615872    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbrm\" (UniqueName: \"kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm\") pod \"payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.618621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.618556    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.634684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.634633    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbrm\" (UniqueName: \"kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm\") pod \"payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.799980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.799941    5074 generic.go:356] "Generic (PLEG): container finished" podID="4e13b562-d3c7-484f-a114-d33a6f5cd895" containerID="b315bed25e05991485f7a66d819f2cd651c99050ee8607b67c7d23f3156ebfd5" exitCode=0
Nov 04 00:00:24.800172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.800080    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs" event={"ID":"4e13b562-d3c7-484f-a114-d33a6f5cd895","Type":"ContainerDied","Data":"b315bed25e05991485f7a66d819f2cd651c99050ee8607b67c7d23f3156ebfd5"}
Nov 04 00:00:24.841855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.841772    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:24.947222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:24.947193    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"]
Nov 04 00:00:25.046948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.046913    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9555ad04-1dc8-4a33-b8d1-c17feae69865" path="/var/lib/kubelet/pods/9555ad04-1dc8-4a33-b8d1-c17feae69865/volumes"
Nov 04 00:00:25.047867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.047845    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fab351-0452-48f0-9d68-260c672a0e85" path="/var/lib/kubelet/pods/a3fab351-0452-48f0-9d68-260c672a0e85/volumes"
Nov 04 00:00:25.048616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.048594    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34218e8-bb4f-417b-9889-0fe3a0a5b486" path="/var/lib/kubelet/pods/c34218e8-bb4f-417b-9889-0fe3a0a5b486/volumes"
Nov 04 00:00:25.158875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.158845    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"]
Nov 04 00:00:25.159089 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:25.159067    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735b69d5_f762_4c20_b6c0_45c16656e2ac.slice/crio-fa2387e91f8391c2e70d4dd0f1adb9b751fbb615fa8850ad0d56f5c9cbd73c01 WatchSource:0}: Error finding container fa2387e91f8391c2e70d4dd0f1adb9b751fbb615fa8850ad0d56f5c9cbd73c01: Status 404 returned error can't find the container with id fa2387e91f8391c2e70d4dd0f1adb9b751fbb615fa8850ad0d56f5c9cbd73c01
Nov 04 00:00:25.811369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.811327    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" event={"ID":"4b1131b3-d78d-4434-9d46-8c2044aa02db","Type":"ContainerStarted","Data":"b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a"}
Nov 04 00:00:25.811992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.811410    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" event={"ID":"4b1131b3-d78d-4434-9d46-8c2044aa02db","Type":"ContainerStarted","Data":"e56086782bc96db365d1a4a7b00a8a81f1dd00dcf8774ce7b6429b6ed825cf06"}
Nov 04 00:00:25.812336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.812311    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:25.814329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.814298    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642" event={"ID":"735b69d5-f762-4c20-b6c0-45c16656e2ac","Type":"ContainerStarted","Data":"e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c"}
Nov 04 00:00:25.814434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.814352    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642" event={"ID":"735b69d5-f762-4c20-b6c0-45c16656e2ac","Type":"ContainerStarted","Data":"fa2387e91f8391c2e70d4dd0f1adb9b751fbb615fa8850ad0d56f5c9cbd73c01"}
Nov 04 00:00:25.815858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.814487    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:25.832251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.832191    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" podStartSLOduration=1.832174204 podStartE2EDuration="1.832174204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:25.831758052 +0000 UTC m=+1890.931358187" watchObservedRunningTime="2026-11-04 00:00:25.832174204 +0000 UTC m=+1890.931774356"
Nov 04 00:00:25.855706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:25.855648    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642" podStartSLOduration=1.855633009 podStartE2EDuration="1.855633009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:25.851474927 +0000 UTC m=+1890.951075071" watchObservedRunningTime="2026-11-04 00:00:25.855633009 +0000 UTC m=+1890.955233153"
Nov 04 00:00:26.031852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.031814    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:26.162167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.162069    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca\") pod \"4e13b562-d3c7-484f-a114-d33a6f5cd895\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") "
Nov 04 00:00:26.162343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.162249    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf\" (UniqueName: \"kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf\") pod \"4e13b562-d3c7-484f-a114-d33a6f5cd895\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") "
Nov 04 00:00:26.162343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.162304    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca\") pod \"4e13b562-d3c7-484f-a114-d33a6f5cd895\" (UID: \"4e13b562-d3c7-484f-a114-d33a6f5cd895\") "
Nov 04 00:00:26.163325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.163295    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca" pod "4e13b562-d3c7-484f-a114-d33a6f5cd895" (UID: "4e13b562-d3c7-484f-a114-d33a6f5cd895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:26.167207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.167180    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca" pod "4e13b562-d3c7-484f-a114-d33a6f5cd895" (UID: "4e13b562-d3c7-484f-a114-d33a6f5cd895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:26.202146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.202110    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf" pod "4e13b562-d3c7-484f-a114-d33a6f5cd895" (UID: "4e13b562-d3c7-484f-a114-d33a6f5cd895"). InnerVolumeSpecName "kube-api-access-2dqpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:26.264881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.264839    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dqpf\" (UniqueName: \"kubernetes.io/projected/4e13b562-d3c7-484f-a114-d33a6f5cd895-kube-api-access-2dqpf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:26.264881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.264883    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:26.265110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.264901    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e13b562-d3c7-484f-a114-d33a6f5cd895-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:26.838208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.838172    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"]
Nov 04 00:00:26.854382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.852607    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"
Nov 04 00:00:26.854382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.853094    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs" event={"ID":"4e13b562-d3c7-484f-a114-d33a6f5cd895","Type":"ContainerDied","Data":"4f645e36b0045e88b10284d764dae80939dc293bf1e135afda0ad0334db61079"}
Nov 04 00:00:26.854382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:26.853142    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f645e36b0045e88b10284d764dae80939dc293bf1e135afda0ad0334db61079"
Nov 04 00:00:27.866431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.866391    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"]
Nov 04 00:00:27.867588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.867562    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4e13b562-d3c7-484f-a114-d33a6f5cd895" containerName="revoke-keys"
Nov 04 00:00:27.867791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.867773    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e13b562-d3c7-484f-a114-d33a6f5cd895" containerName="revoke-keys"
Nov 04 00:00:27.868224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.868203    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e13b562-d3c7-484f-a114-d33a6f5cd895" containerName="revoke-keys"
Nov 04 00:00:27.872412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.872392    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:27.873022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.872990    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"]
Nov 04 00:00:27.879909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.879883    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:00:27.885651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.884320    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:00:27.915765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.907060    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:27.915765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.907272    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5zv\" (UniqueName: \"kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:27.915765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:27.907374    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.009246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.009176    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5zv\" (UniqueName: \"kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.009499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.009339    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.009499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.009450    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.010443 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.010411    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.025412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.025383    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5zv\" (UniqueName: \"kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.039754 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:28.039532    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:00:28.041463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.041433    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85c84efd-hmt8h\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.195387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.195341    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:28.500852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.498311    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"]
Nov 04 00:00:28.906521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:28.906484    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h" event={"ID":"38ae30cd-734b-4c8d-82c8-716b83cbca4e","Type":"ContainerStarted","Data":"124877fe38e9c282b711ac1f752c12c01cc3ff8c2310291515ac52d1857be9aa"}
Nov 04 00:00:29.718090 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:29.718046    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:29.921714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:29.921676    5074 generic.go:356] "Generic (PLEG): container finished" podID="38ae30cd-734b-4c8d-82c8-716b83cbca4e" containerID="4b6279783e4d407e930f62ea2553490efeeaa1d3f662e6da1b22bf5e3fc686eb" exitCode=0
Nov 04 00:00:29.922184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:29.921773    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h" event={"ID":"38ae30cd-734b-4c8d-82c8-716b83cbca4e","Type":"ContainerDied","Data":"4b6279783e4d407e930f62ea2553490efeeaa1d3f662e6da1b22bf5e3fc686eb"}
Nov 04 00:00:30.429010 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:30.428969    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:30.725970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:30.725925    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9" podUID="eee25e7c-e20b-406d-9b79-8f552b0e639e" containerName="main" containerID="cri-o://ec1c4bc71d4c716c2c5b3c31fbe83c6f4c057c7ee8ce7e97e14b45ddded0869b" gracePeriod=45
Nov 04 00:00:30.960957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:30.960917    5074 generic.go:356] "Generic (PLEG): container finished" podID="eee25e7c-e20b-406d-9b79-8f552b0e639e" containerID="ec1c4bc71d4c716c2c5b3c31fbe83c6f4c057c7ee8ce7e97e14b45ddded0869b" exitCode=0
Nov 04 00:00:30.961459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:30.960971    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9" event={"ID":"eee25e7c-e20b-406d-9b79-8f552b0e639e","Type":"ContainerDied","Data":"ec1c4bc71d4c716c2c5b3c31fbe83c6f4c057c7ee8ce7e97e14b45ddded0869b"}
Nov 04 00:00:31.061833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.061801    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 04 00:00:31.190017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.189979    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache\") pod \"eee25e7c-e20b-406d-9b79-8f552b0e639e\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") "
Nov 04 00:00:31.190188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190038    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir\") pod \"eee25e7c-e20b-406d-9b79-8f552b0e639e\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") "
Nov 04 00:00:31.190188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190081    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs\" (UniqueName: \"kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs\") pod \"eee25e7c-e20b-406d-9b79-8f552b0e639e\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") "
Nov 04 00:00:31.190320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190296    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache" pod "eee25e7c-e20b-406d-9b79-8f552b0e639e" (UID: "eee25e7c-e20b-406d-9b79-8f552b0e639e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:31.190320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190308    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir" pod "eee25e7c-e20b-406d-9b79-8f552b0e639e" (UID: "eee25e7c-e20b-406d-9b79-8f552b0e639e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:31.190432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190309    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5\" (UniqueName: \"kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5\") pod \"eee25e7c-e20b-406d-9b79-8f552b0e639e\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") "
Nov 04 00:00:31.190487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190463    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home\") pod \"eee25e7c-e20b-406d-9b79-8f552b0e639e\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") "
Nov 04 00:00:31.190534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.190510    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm\") pod \"eee25e7c-e20b-406d-9b79-8f552b0e639e\" (UID: \"eee25e7c-e20b-406d-9b79-8f552b0e639e\") "
Nov 04 00:00:31.191066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.191037    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home" pod "eee25e7c-e20b-406d-9b79-8f552b0e639e" (UID: "eee25e7c-e20b-406d-9b79-8f552b0e639e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:31.191407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.191370    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.191407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.191399    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.191567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.191413    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.193153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.193126    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5" pod "eee25e7c-e20b-406d-9b79-8f552b0e639e" (UID: "eee25e7c-e20b-406d-9b79-8f552b0e639e"). InnerVolumeSpecName "kube-api-access-stnw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:31.195715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.195678    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:31.215785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.207879    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs" pod "eee25e7c-e20b-406d-9b79-8f552b0e639e" (UID: "eee25e7c-e20b-406d-9b79-8f552b0e639e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:31.215785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.210858    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm" pod "eee25e7c-e20b-406d-9b79-8f552b0e639e" (UID: "eee25e7c-e20b-406d-9b79-8f552b0e639e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:31.292986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.292903    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv\" (UniqueName: \"kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv\") pod \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") "
Nov 04 00:00:31.293254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.293223    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca\") pod \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") "
Nov 04 00:00:31.293386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.293277    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca\") pod \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\" (UID: \"38ae30cd-734b-4c8d-82c8-716b83cbca4e\") "
Nov 04 00:00:31.293972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.293926    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca" pod "38ae30cd-734b-4c8d-82c8-716b83cbca4e" (UID: "38ae30cd-734b-4c8d-82c8-716b83cbca4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:31.293972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.293930    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee25e7c-e20b-406d-9b79-8f552b0e639e-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.294201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.293986    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnw5\" (UniqueName: \"kubernetes.io/projected/eee25e7c-e20b-406d-9b79-8f552b0e639e-kube-api-access-stnw5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.294201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.294045    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee25e7c-e20b-406d-9b79-8f552b0e639e-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.295676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.295649    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv" pod "38ae30cd-734b-4c8d-82c8-716b83cbca4e" (UID: "38ae30cd-734b-4c8d-82c8-716b83cbca4e"). InnerVolumeSpecName "kube-api-access-ml5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:31.296088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.296063    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca" pod "38ae30cd-734b-4c8d-82c8-716b83cbca4e" (UID: "38ae30cd-734b-4c8d-82c8-716b83cbca4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:31.395320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.395280    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.395320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.395322    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ae30cd-734b-4c8d-82c8-716b83cbca4e-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.395550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.395339    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5zv\" (UniqueName: \"kubernetes.io/projected/38ae30cd-734b-4c8d-82c8-716b83cbca4e-kube-api-access-ml5zv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:31.973149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.973115    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"
Nov 04 00:00:31.973149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.973126    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h" event={"ID":"38ae30cd-734b-4c8d-82c8-716b83cbca4e","Type":"ContainerDied","Data":"124877fe38e9c282b711ac1f752c12c01cc3ff8c2310291515ac52d1857be9aa"}
Nov 04 00:00:31.973714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.973188    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="124877fe38e9c282b711ac1f752c12c01cc3ff8c2310291515ac52d1857be9aa"
Nov 04 00:00:31.975648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.975442    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9" event={"ID":"eee25e7c-e20b-406d-9b79-8f552b0e639e","Type":"ContainerDied","Data":"1c356732f3c184e11e2194eded9e5b0a1ff11a4a74e3468c909bbd9040437b15"}
Nov 04 00:00:31.975648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.975507    5074 scope.go:122] "RemoveContainer" containerID="ec1c4bc71d4c716c2c5b3c31fbe83c6f4c057c7ee8ce7e97e14b45ddded0869b"
Nov 04 00:00:31.976197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:31.976177    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"
Nov 04 00:00:32.027685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:32.027654    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"]
Nov 04 00:00:32.034083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:32.034053    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/auth-test-model-b3bea7-kserve-79fbb85c89-266t9"]
Nov 04 00:00:33.022843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:33.022802    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee25e7c-e20b-406d-9b79-8f552b0e639e" path="/var/lib/kubelet/pods/eee25e7c-e20b-406d-9b79-8f552b0e639e/volumes"
Nov 04 00:00:36.651277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:36.642887    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"]
Nov 04 00:00:36.867760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:36.862603    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:00:37.900764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:37.882178    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:00:38.136997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.136961    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"]
Nov 04 00:00:38.138168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.138143    5074 cpu_manager.go:414] "containerMap: removing container" podUID="38ae30cd-734b-4c8d-82c8-716b83cbca4e" containerName="revoke-keys"
Nov 04 00:00:38.138168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.138168    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ae30cd-734b-4c8d-82c8-716b83cbca4e" containerName="revoke-keys"
Nov 04 00:00:38.138377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.138193    5074 cpu_manager.go:414] "containerMap: removing container" podUID="eee25e7c-e20b-406d-9b79-8f552b0e639e" containerName="main"
Nov 04 00:00:38.138377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.138202    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee25e7c-e20b-406d-9b79-8f552b0e639e" containerName="main"
Nov 04 00:00:38.138377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.138349    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee25e7c-e20b-406d-9b79-8f552b0e639e" containerName="main"
Nov 04 00:00:38.138377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.138375    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ae30cd-734b-4c8d-82c8-716b83cbca4e" containerName="revoke-keys"
Nov 04 00:00:38.147470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.147440    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.151582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.151513    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"]
Nov 04 00:00:38.172339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.172306    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ba68e7\"/\"kube-root-ca.crt\""
Nov 04 00:00:38.172971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.172947    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ba68e7\"/\"noten-model-ba68e7-kserve-self-signed-certs\""
Nov 04 00:00:38.173235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.173217    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ba68e7\"/\"default-dockercfg-pbspc\""
Nov 04 00:00:38.173414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.173397    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ba68e7\"/\"openshift-service-ca.crt\""
Nov 04 00:00:38.339762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.335473    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.339762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.335623    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.339762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.335912    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.339762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.336007    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.339762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.336040    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttzx\" (UniqueName: \"kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.339762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.336087    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.437080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.436047    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"]
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.438616    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.438695    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.438726    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttzx\" (UniqueName: \"kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.438789    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.438883    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.438934    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.443476    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.460764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.443725    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.462061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.462028    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.491792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.472082    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"]
Nov 04 00:00:38.491792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.472211    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.491792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.474751    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttzx\" (UniqueName: \"kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.491792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.477596    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:00:38.491792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.477843    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:00:38.501799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.495220    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.530629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.530549    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs\") pod \"noten-model-ba68e7-kserve-68674f8d5c-sqwmj\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") " pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.540841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.540681    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.540841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.540726    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.540841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.540843    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rff\" (UniqueName: \"kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.642793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.642716    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.643004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.642801    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.643004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.642921    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rff\" (UniqueName: \"kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.644293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.644264    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.647100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.647073    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.656986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.656959    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rff\" (UniqueName: \"kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff\") pod \"maas-api-revoke-sub-78e2c9ce-lhcrs\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:38.784813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.782930    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:38.866266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:38.865957    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:39.136383 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:39.136353    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65b2acae_1a0f_4d9f_a643_08701153e9d9.slice/crio-6467c332610d37253eabad53bf999f9271fe134b2a75fca9100b8716f6095434 WatchSource:0}: Error finding container 6467c332610d37253eabad53bf999f9271fe134b2a75fca9100b8716f6095434: Status 404 returned error can't find the container with id 6467c332610d37253eabad53bf999f9271fe134b2a75fca9100b8716f6095434
Nov 04 00:00:39.136846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:39.136486    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"]
Nov 04 00:00:39.156824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:39.156790    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj" event={"ID":"65b2acae-1a0f-4d9f-a643-08701153e9d9","Type":"ContainerStarted","Data":"6467c332610d37253eabad53bf999f9271fe134b2a75fca9100b8716f6095434"}
Nov 04 00:00:39.252055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:39.244386    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"]
Nov 04 00:00:39.283901 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:39.283866    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed9152e_ee12_4c23_97d8_c2548cb6d695.slice/crio-e57ce51dffc7aeb88d6e9c659678353e1aab4544b7c0d41dc8c9741ade1f4e14 WatchSource:0}: Error finding container e57ce51dffc7aeb88d6e9c659678353e1aab4544b7c0d41dc8c9741ade1f4e14: Status 404 returned error can't find the container with id e57ce51dffc7aeb88d6e9c659678353e1aab4544b7c0d41dc8c9741ade1f4e14
Nov 04 00:00:40.168406 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:40.168189    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj" event={"ID":"65b2acae-1a0f-4d9f-a643-08701153e9d9","Type":"ContainerStarted","Data":"7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3"}
Nov 04 00:00:40.171027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:40.170995    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs" event={"ID":"9ed9152e-ee12-4c23-97d8-c2548cb6d695","Type":"ContainerStarted","Data":"594c2e6e84af8f7dd9ea6655b5779feb95116afb6b70ff59d6c225173f2fcaf4"}
Nov 04 00:00:40.171175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:40.171046    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs" event={"ID":"9ed9152e-ee12-4c23-97d8-c2548cb6d695","Type":"ContainerStarted","Data":"e57ce51dffc7aeb88d6e9c659678353e1aab4544b7c0d41dc8c9741ade1f4e14"}
Nov 04 00:00:40.198402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:40.198337    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj" podStartSLOduration=2.198318287 podStartE2EDuration="2.198318287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:40.196061493 +0000 UTC m=+1905.295661638" watchObservedRunningTime="2026-11-04 00:00:40.198318287 +0000 UTC m=+1905.297918432"
Nov 04 00:00:40.607313 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:40.607271    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed9152e_ee12_4c23_97d8_c2548cb6d695.slice/crio-594c2e6e84af8f7dd9ea6655b5779feb95116afb6b70ff59d6c225173f2fcaf4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed9152e_ee12_4c23_97d8_c2548cb6d695.slice/crio-conmon-594c2e6e84af8f7dd9ea6655b5779feb95116afb6b70ff59d6c225173f2fcaf4.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:40.785590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:40.785508    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:41.187239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:41.187205    5074 generic.go:356] "Generic (PLEG): container finished" podID="9ed9152e-ee12-4c23-97d8-c2548cb6d695" containerID="594c2e6e84af8f7dd9ea6655b5779feb95116afb6b70ff59d6c225173f2fcaf4" exitCode=0
Nov 04 00:00:41.187769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:41.187296    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs" event={"ID":"9ed9152e-ee12-4c23-97d8-c2548cb6d695","Type":"ContainerDied","Data":"594c2e6e84af8f7dd9ea6655b5779feb95116afb6b70ff59d6c225173f2fcaf4"}
Nov 04 00:00:41.870538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:41.870249    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px" podUID="4f0cca6f-ea89-4754-8eed-3d8069713fd1" containerName="main" containerID="cri-o://00899d3a26c84ddd0d415a0b4bd71b8ef407fad39aa2bb951f289224141cab64" gracePeriod=45
Nov 04 00:00:41.932648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:41.932594    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"]
Nov 04 00:00:42.310227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.309272    5074 generic.go:356] "Generic (PLEG): container finished" podID="4f0cca6f-ea89-4754-8eed-3d8069713fd1" containerID="00899d3a26c84ddd0d415a0b4bd71b8ef407fad39aa2bb951f289224141cab64" exitCode=0
Nov 04 00:00:42.310227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.309886    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px" event={"ID":"4f0cca6f-ea89-4754-8eed-3d8069713fd1","Type":"ContainerDied","Data":"00899d3a26c84ddd0d415a0b4bd71b8ef407fad39aa2bb951f289224141cab64"}
Nov 04 00:00:42.356985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.356798    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387378    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache\") pod \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") "
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387448    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk\" (UniqueName: \"kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk\") pod \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") "
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387539    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home\") pod \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") "
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387686    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir\") pod \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") "
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387716    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs\") pod \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") "
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387889    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache" pod "4f0cca6f-ea89-4754-8eed-3d8069713fd1" (UID: "4f0cca6f-ea89-4754-8eed-3d8069713fd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.387946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm\") pod \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\" (UID: \"4f0cca6f-ea89-4754-8eed-3d8069713fd1\") "
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.388176    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir" pod "4f0cca6f-ea89-4754-8eed-3d8069713fd1" (UID: "4f0cca6f-ea89-4754-8eed-3d8069713fd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.388916    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.388940    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.390314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.389454    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home" pod "4f0cca6f-ea89-4754-8eed-3d8069713fd1" (UID: "4f0cca6f-ea89-4754-8eed-3d8069713fd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:42.395712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.395255    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm" pod "4f0cca6f-ea89-4754-8eed-3d8069713fd1" (UID: "4f0cca6f-ea89-4754-8eed-3d8069713fd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:42.395712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.395696    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs" pod "4f0cca6f-ea89-4754-8eed-3d8069713fd1" (UID: "4f0cca6f-ea89-4754-8eed-3d8069713fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:42.410002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.409881    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk" pod "4f0cca6f-ea89-4754-8eed-3d8069713fd1" (UID: "4f0cca6f-ea89-4754-8eed-3d8069713fd1"). InnerVolumeSpecName "kube-api-access-w4pnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:42.497692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.497660    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0cca6f-ea89-4754-8eed-3d8069713fd1-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.497692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.497695    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.497969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.497711    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4pnk\" (UniqueName: \"kubernetes.io/projected/4f0cca6f-ea89-4754-8eed-3d8069713fd1-kube-api-access-w4pnk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.497969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.497727    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0cca6f-ea89-4754-8eed-3d8069713fd1-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.498266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.498240    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:42.598930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.598891    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca\") pod \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") "
Nov 04 00:00:42.599110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.599011    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca\") pod \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") "
Nov 04 00:00:42.599110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.599086    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff\" (UniqueName: \"kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff\") pod \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\" (UID: \"9ed9152e-ee12-4c23-97d8-c2548cb6d695\") "
Nov 04 00:00:42.599376 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.599348    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca" pod "9ed9152e-ee12-4c23-97d8-c2548cb6d695" (UID: "9ed9152e-ee12-4c23-97d8-c2548cb6d695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:42.599715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.599692    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.601249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.601224    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff" pod "9ed9152e-ee12-4c23-97d8-c2548cb6d695" (UID: "9ed9152e-ee12-4c23-97d8-c2548cb6d695"). InnerVolumeSpecName "kube-api-access-n8rff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:42.601489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.601469    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca" pod "9ed9152e-ee12-4c23-97d8-c2548cb6d695" (UID: "9ed9152e-ee12-4c23-97d8-c2548cb6d695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:42.701181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.701142    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9152e-ee12-4c23-97d8-c2548cb6d695-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.701181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:42.701171    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rff\" (UniqueName: \"kubernetes.io/projected/9ed9152e-ee12-4c23-97d8-c2548cb6d695-kube-api-access-n8rff\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:43.319386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.319303    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"
Nov 04 00:00:43.319386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.319339    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px" event={"ID":"4f0cca6f-ea89-4754-8eed-3d8069713fd1","Type":"ContainerDied","Data":"1cc3ed2c5c4fdbb93b7e6a8f4fa705a28ba3b28099705a703b79d03db1c52e8f"}
Nov 04 00:00:43.319951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.319395    5074 scope.go:122] "RemoveContainer" containerID="00899d3a26c84ddd0d415a0b4bd71b8ef407fad39aa2bb951f289224141cab64"
Nov 04 00:00:43.321333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.321311    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"
Nov 04 00:00:43.321333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.321316    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs" event={"ID":"9ed9152e-ee12-4c23-97d8-c2548cb6d695","Type":"ContainerDied","Data":"e57ce51dffc7aeb88d6e9c659678353e1aab4544b7c0d41dc8c9741ade1f4e14"}
Nov 04 00:00:43.321544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.321364    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e57ce51dffc7aeb88d6e9c659678353e1aab4544b7c0d41dc8c9741ade1f4e14"
Nov 04 00:00:43.342143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.342106    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"]
Nov 04 00:00:43.346175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:43.346150    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/auth-test-model-58b75d-kserve-58556bb75c-hs7px"]
Nov 04 00:00:44.620845 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:44.620804    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:45.022008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:45.021965    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0cca6f-ea89-4754-8eed-3d8069713fd1" path="/var/lib/kubelet/pods/4f0cca6f-ea89-4754-8eed-3d8069713fd1/volumes"
Nov 04 00:00:46.737695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.737660    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7302-19824e"]
Nov 04 00:00:46.738601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.738578    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4f0cca6f-ea89-4754-8eed-3d8069713fd1" containerName="main"
Nov 04 00:00:46.738601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.738602    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0cca6f-ea89-4754-8eed-3d8069713fd1" containerName="main"
Nov 04 00:00:46.738905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.738619    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9ed9152e-ee12-4c23-97d8-c2548cb6d695" containerName="revoke-keys"
Nov 04 00:00:46.738905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.738628    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed9152e-ee12-4c23-97d8-c2548cb6d695" containerName="revoke-keys"
Nov 04 00:00:46.739018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.738961    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0cca6f-ea89-4754-8eed-3d8069713fd1" containerName="main"
Nov 04 00:00:46.739018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.738982    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed9152e-ee12-4c23-97d8-c2548cb6d695" containerName="revoke-keys"
Nov 04 00:00:46.742691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.742668    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:46.764863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.748967    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hxqlx\""
Nov 04 00:00:46.770522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.770474    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7302-19824e"]
Nov 04 00:00:46.906503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:46.906465    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp4nh\" (UniqueName: \"kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh\") pod \"test-curl-7302-19824e\" (UID: \"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2\") " pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:47.007982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:47.007902    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp4nh\" (UniqueName: \"kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh\") pod \"test-curl-7302-19824e\" (UID: \"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2\") " pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:47.021097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:47.021071    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp4nh\" (UniqueName: \"kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh\") pod \"test-curl-7302-19824e\" (UID: \"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2\") " pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:47.068276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:47.068166    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:47.431801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:47.427610    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7302-19824e"]
Nov 04 00:00:47.433579 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:47.433542    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0bfc696_d2f2_47ed_ba5a_ad76200fa8c2.slice/crio-ded5e4c0fe5d0e78eabd616edd5f407c01f04198eb34daa68469ee8323cb2c62 WatchSource:0}: Error finding container ded5e4c0fe5d0e78eabd616edd5f407c01f04198eb34daa68469ee8323cb2c62: Status 404 returned error can't find the container with id ded5e4c0fe5d0e78eabd616edd5f407c01f04198eb34daa68469ee8323cb2c62
Nov 04 00:00:48.369563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:48.369521    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7302-19824e" event={"ID":"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2","Type":"ContainerStarted","Data":"042f137ee3b0847ee8d2f0cd3d25b5145fbf4cec4703ab3796762182d639bbcf"}
Nov 04 00:00:48.369989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:48.369573    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7302-19824e" event={"ID":"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2","Type":"ContainerStarted","Data":"ded5e4c0fe5d0e78eabd616edd5f407c01f04198eb34daa68469ee8323cb2c62"}
Nov 04 00:00:48.387151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:48.387102    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7302-19824e" podStartSLOduration=2.387089608 podStartE2EDuration="2.387089608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:48.386809615 +0000 UTC m=+1913.486409758" watchObservedRunningTime="2026-11-04 00:00:48.387089608 +0000 UTC m=+1913.486689751"
Nov 04 00:00:48.784861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:48.784821    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:48.797549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:48.797524    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:49.068839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:49.066501    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"]
Nov 04 00:00:49.155069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:49.155032    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7302-19824e"]
Nov 04 00:00:49.168351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:49.167846    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7302-19824e"]
Nov 04 00:00:49.442771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:49.439131    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:00:49.480762 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:49.479945    5074 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-self-signed-certs: secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:49.480762 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:49.480027    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs podName:65b2acae-1a0f-4d9f-a643-08701153e9d9 nodeName:}" failed. No retries permitted until 2026-11-04 00:00:49.980006343 +0000 UTC m=+1915.079606486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs") pod "noten-model-ba68e7-kserve-68674f8d5c-sqwmj" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9") : secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:49.987520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:49.987487    5074 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-self-signed-certs: secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:49.987702 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:49.987558    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs podName:65b2acae-1a0f-4d9f-a643-08701153e9d9 nodeName:}" failed. No retries permitted until 2026-11-04 00:00:50.987543399 +0000 UTC m=+1916.087143525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs") pod "noten-model-ba68e7-kserve-68674f8d5c-sqwmj" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9") : secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:50.437813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:50.432088    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7302-19824e" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" containerName="test-curl-7302-19824e" containerID="cri-o://042f137ee3b0847ee8d2f0cd3d25b5145fbf4cec4703ab3796762182d639bbcf" gracePeriod=2
Nov 04 00:00:50.437813 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:50.436613    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7302-19824e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:50.460538 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:50.460494    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7302-19824e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:50.704685 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:50.704594    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa3249_0fc9_4c24_93fd_c51e3dccc018.slice/crio-c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:00:51.001493 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:51.001406    5074 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-self-signed-certs: secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:51.001675 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:51.001496    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs podName:65b2acae-1a0f-4d9f-a643-08701153e9d9 nodeName:}" failed. No retries permitted until 2026-11-04 00:00:53.001475876 +0000 UTC m=+1918.101076001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs") pod "noten-model-ba68e7-kserve-68674f8d5c-sqwmj" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9") : secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:51.689178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:51.689110    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" podUID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" containerName="main" containerID="cri-o://e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da" gracePeriod=45
Nov 04 00:00:51.772467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:51.772426    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" podUID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": dial tcp 10.132.0.192:8000: connect: connection refused"
Nov 04 00:00:52.063152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.063121    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 04 00:00:52.138687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.137561    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache\") pod \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") "
Nov 04 00:00:52.138687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.137631    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl\" (UniqueName: \"kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl\") pod \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") "
Nov 04 00:00:52.143040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.137730    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir\") pod \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") "
Nov 04 00:00:52.143200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.143157    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home\") pod \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") "
Nov 04 00:00:52.143284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.143252    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs\") pod \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") "
Nov 04 00:00:52.143345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.143318    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm\") pod \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\" (UID: \"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328\") "
Nov 04 00:00:52.145052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.144685    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir" pod "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" (UID: "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:52.145052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.144912    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache" pod "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" (UID: "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:52.153451 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.148962    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home" pod "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" (UID: "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:52.154665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.154637    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl" pod "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" (UID: "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328"). InnerVolumeSpecName "kube-api-access-q5xcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:52.158477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.158346    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm" pod "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" (UID: "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:52.158966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.158898    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs" pod "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" (UID: "f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:52.245954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.245907    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:52.245954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.245949    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:52.246197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.245962    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:52.246197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.245975    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5xcl\" (UniqueName: \"kubernetes.io/projected/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-kube-api-access-q5xcl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:52.246197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.245990    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:52.246197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.246000    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:52.499787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.499721    5074 generic.go:356] "Generic (PLEG): container finished" podID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" containerID="042f137ee3b0847ee8d2f0cd3d25b5145fbf4cec4703ab3796762182d639bbcf" exitCode=137
Nov 04 00:00:52.501548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.501520    5074 generic.go:356] "Generic (PLEG): container finished" podID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" containerID="e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da" exitCode=0
Nov 04 00:00:52.501674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.501579    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"
Nov 04 00:00:52.501674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.501575    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" event={"ID":"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328","Type":"ContainerDied","Data":"e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da"}
Nov 04 00:00:52.501816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.501691    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc" event={"ID":"f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328","Type":"ContainerDied","Data":"71d2331d0750a6b0e0727dae932c4e79d5c557899907c7ec051459aaa7b8f973"}
Nov 04 00:00:52.501816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.501714    5074 scope.go:122] "RemoveContainer" containerID="e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da"
Nov 04 00:00:52.521930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.519853    5074 scope.go:122] "RemoveContainer" containerID="e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da"
Nov 04 00:00:52.523005 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:52.522971    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da\": container with ID starting with e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da not found: ID does not exist" containerID="e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da"
Nov 04 00:00:52.523139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.523014    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da"} err="failed to get container status \"e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da\": rpc error: code = NotFound desc = could not find container \"e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da\": container with ID starting with e3d69b9a44b99a3217ff2e01caeb0942b820c75462b1a5ee47ce08b3d95a58da not found: ID does not exist"
Nov 04 00:00:52.565769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.565443    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"]
Nov 04 00:00:52.565769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.565498    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/auth-test-model-8093f5-kserve-589f895997-hszmc"]
Nov 04 00:00:52.922601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:52.922572    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:53.001683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.001589    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh\" (UniqueName: \"kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh\") pod \"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2\" (UID: \"e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2\") "
Nov 04 00:00:53.002368 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:53.002339    5074 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-self-signed-certs: secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:53.002502 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:53.002415    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs podName:65b2acae-1a0f-4d9f-a643-08701153e9d9 nodeName:}" failed. No retries permitted until 2026-11-04 00:00:57.002396784 +0000 UTC m=+1922.101996905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs") pod "noten-model-ba68e7-kserve-68674f8d5c-sqwmj" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9") : secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:53.004337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.004302    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh" pod "e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" (UID: "e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2"). InnerVolumeSpecName "kube-api-access-sp4nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:53.027805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.027774    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" path="/var/lib/kubelet/pods/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2/volumes"
Nov 04 00:00:53.028474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.028438    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" path="/var/lib/kubelet/pods/f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328/volumes"
Nov 04 00:00:53.107477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.107438    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp4nh\" (UniqueName: \"kubernetes.io/projected/e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2-kube-api-access-sp4nh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:53.117715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.117679    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"]
Nov 04 00:00:53.119149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.119117    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" containerName="test-curl-7302-19824e"
Nov 04 00:00:53.119332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.119307    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" containerName="test-curl-7302-19824e"
Nov 04 00:00:53.119466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.119454    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" containerName="main"
Nov 04 00:00:53.119576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.119557    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" containerName="main"
Nov 04 00:00:53.119861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.119843    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3bdf137-4a6e-4ca0-a3ee-0b7a3d269328" containerName="main"
Nov 04 00:00:53.119945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.119884    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0bfc696-d2f2-47ed-ba5a-ad76200fa8c2" containerName="test-curl-7302-19824e"
Nov 04 00:00:53.125007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.124782    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"]
Nov 04 00:00:53.125007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.124902    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.131551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.131529    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8093f5-openshift-default-dockercfg-fkkkn\""
Nov 04 00:00:53.208349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208308    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.208530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208486    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.208530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208519    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.208786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208720    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.208873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208800    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.208873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208836    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4xd\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.208978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.208898    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.209056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.209036    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.209104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.209076    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.311053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.310515    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.311334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.311306    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.311475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.311459    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4xd\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.311588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.311576    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.311811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.311792    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.311953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.311935    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.312540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.312509    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.312673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.312592    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.312799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.312756    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.312871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.312803    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.323876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.313170    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.332103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.332062    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.332368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.332339    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.334420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.334394    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.336074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.336043    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.352345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.352309    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.352675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.352650    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4xd\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.355933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.355908    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.454552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.454511    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:53.524763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.524006    5074 scope.go:122] "RemoveContainer" containerID="042f137ee3b0847ee8d2f0cd3d25b5145fbf4cec4703ab3796762182d639bbcf"
Nov 04 00:00:53.524763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.524171    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7302-19824e"
Nov 04 00:00:53.672517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.672480    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"]
Nov 04 00:00:53.678308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.678282    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:53.681494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.681468    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"]
Nov 04 00:00:53.686158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.686136    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:53.717994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.704985    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-27f8a1\""
Nov 04 00:00:53.717994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.707653    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"]
Nov 04 00:00:53.717994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.715878    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"]
Nov 04 00:00:53.776839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.776810    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-27f8a1-dockercfg-2d2n9\""
Nov 04 00:00:53.823026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.822941    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"]
Nov 04 00:00:53.824514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.824484    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzvt\" (UniqueName: \"kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt\") pod \"payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") " pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:53.824869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.824841    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:53.825325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.825209    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume\") pod \"payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") " pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:53.825694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.825672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9995b\" (UniqueName: \"kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b\") pod \"payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:53.860690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.860642    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:00:53.860954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.860931    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:00:53.861173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.861153    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:00:53.934506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.931969    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:53.934506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.932075    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume\") pod \"payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") " pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:53.934506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.932202    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9995b\" (UniqueName: \"kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b\") pod \"payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:53.934506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.932319    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzvt\" (UniqueName: \"kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt\") pod \"payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") " pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:53.934506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.933476    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:53.934506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.934225    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume\") pod \"payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") " pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:53.966055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:53.965939    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzvt\" (UniqueName: \"kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt\") pod \"payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") " pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:54.001467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.000944    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9995b\" (UniqueName: \"kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b\") pod \"payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:54.047981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.047862    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:54.294246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.294215    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:54.376815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.375344    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"]
Nov 04 00:00:54.551434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.551394    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" event={"ID":"975814af-8159-4623-b09e-44c15fb2ddc1","Type":"ContainerStarted","Data":"d477700ee33b9c250ab34bab6400b2cc8f5cf5215c159a1721db207cd1e27f39"}
Nov 04 00:00:54.551600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.551454    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" event={"ID":"975814af-8159-4623-b09e-44c15fb2ddc1","Type":"ContainerStarted","Data":"432d79b0359f011dc2daff9cbe58be5458d51c4c842045adb811e15f0f868942"}
Nov 04 00:00:54.555350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.555316    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" event={"ID":"424cc865-b851-4e2b-ab02-64387e84b727","Type":"ContainerStarted","Data":"b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d"}
Nov 04 00:00:54.555458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.555377    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" event={"ID":"424cc865-b851-4e2b-ab02-64387e84b727","Type":"ContainerStarted","Data":"de88696c3b8f15a6f88290f45d7585e3ec458ae4de676bb5cefc7c0e97d14ba2"}
Nov 04 00:00:54.555757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.555721    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:00:54.580133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.580069    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" podStartSLOduration=1.580050365 podStartE2EDuration="1.580050365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:54.576309527 +0000 UTC m=+1919.675909672" watchObservedRunningTime="2026-11-04 00:00:54.580050365 +0000 UTC m=+1919.679650511"
Nov 04 00:00:54.613518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.612933    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podStartSLOduration=1.612915424 podStartE2EDuration="1.612915424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:54.611805423 +0000 UTC m=+1919.711405573" watchObservedRunningTime="2026-11-04 00:00:54.612915424 +0000 UTC m=+1919.712515769"
Nov 04 00:00:54.628497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:54.628468    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"]
Nov 04 00:00:55.454902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.454869    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:00:55.456483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.456452    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:00:55.456654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.456509    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:00:55.566225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.566184    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b" event={"ID":"a87e1087-c0d8-43a8-8566-6584e5c3c1c7","Type":"ContainerStarted","Data":"c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda"}
Nov 04 00:00:55.566726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.566702    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:00:55.566907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.566760    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b" event={"ID":"a87e1087-c0d8-43a8-8566-6584e5c3c1c7","Type":"ContainerStarted","Data":"4c35caa966f81abed43b49310765fb466321bb953b83cb636715106180fc2a5f"}
Nov 04 00:00:55.589564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:55.589497    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b" podStartSLOduration=2.58948059 podStartE2EDuration="2.58948059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:55.588842573 +0000 UTC m=+1920.688442718" watchObservedRunningTime="2026-11-04 00:00:55.58948059 +0000 UTC m=+1920.689080736"
Nov 04 00:00:56.455048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.455014    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:00:56.455638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.455605    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:00:56.743348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.743266    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"]
Nov 04 00:00:56.757392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.757357    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"]
Nov 04 00:00:56.757728 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.757699    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:56.761024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.760996    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"]
Nov 04 00:00:56.770033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.770010    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:56.771163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.771143    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3a002b\""
Nov 04 00:00:56.774691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.774612    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"]
Nov 04 00:00:56.795017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.794979    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3a002b-dockercfg-mpxkx\""
Nov 04 00:00:56.904761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.902086    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2b4h\" (UniqueName: \"kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h\") pod \"payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:56.904761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.902159    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:56.904761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.902446    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmck\" (UniqueName: \"kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck\") pod \"payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:56.904761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.902488    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:56.967783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:56.966873    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" podUID="fb75763c-f905-4f8b-9456-6a18f512f2a5" containerName="main" containerID="cri-o://a1284f77b0b351d549fdaf3e253b2d244f6ecca98d6413068aeea655c6ab9f06" gracePeriod=45
Nov 04 00:00:57.048398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.046961    5074 scope.go:122] "RemoveContainer" containerID="f2a4849e5f6623c86bef04a59bacb7263ec9781c274f8d2c888646f58fdd17a3"
Nov 04 00:00:57.075893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.075852    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2b4h\" (UniqueName: \"kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h\") pod \"payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:57.076509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.076484    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:57.077009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.076987    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmck\" (UniqueName: \"kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck\") pod \"payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:57.077195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.077177    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:57.084852 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:57.077621    5074 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-self-signed-certs: secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:57.121812 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:57.106132    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs podName:65b2acae-1a0f-4d9f-a643-08701153e9d9 nodeName:}" failed. No retries permitted until 2026-11-04 00:01:05.106099035 +0000 UTC m=+1930.205699194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs") pod "noten-model-ba68e7-kserve-68674f8d5c-sqwmj" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9") : secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:00:57.121812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.080578    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:57.121812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.079626    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:57.169227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.137789    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2b4h\" (UniqueName: \"kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h\") pod \"payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:57.176922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.176525    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmck\" (UniqueName: \"kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck\") pod \"payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:57.197320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.197271    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" podUID="fb75763c-f905-4f8b-9456-6a18f512f2a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": dial tcp 10.132.0.186:8000: connect: connection refused"
Nov 04 00:00:57.247481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.247450    5074 scope.go:122] "RemoveContainer" containerID="5264b6660ce40b8c7d0dedf730c57eaeb20dd879a4f026b04a14d2b15646e7a4"
Nov 04 00:00:57.258991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.258664    5074 scope.go:122] "RemoveContainer" containerID="236c67f8dba143ecf5e619f035533b2a1eb9d1e6a8b8a592f2ca1e8efe814730"
Nov 04 00:00:57.269509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.269485    5074 scope.go:122] "RemoveContainer" containerID="3aa5950f71ee29820cd61e579d2be31673b8dcdba03e05f8854341a0b0f5712f"
Nov 04 00:00:57.281501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.281482    5074 scope.go:122] "RemoveContainer" containerID="6c05217cd657cfa78c2a50edcf3c07fd10bf2ead218ba158cb056bd0f3f177fe"
Nov 04 00:00:57.295104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.295078    5074 scope.go:122] "RemoveContainer" containerID="2eaf4124de543d94e06f63137f7437aabef3615165468e513de39f9e4ac6fe74"
Nov 04 00:00:57.383454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.383432    5074 scope.go:122] "RemoveContainer" containerID="99f110a19b87cd667b32cfb5bab50098ff6b5e70477b204a76045d930df8a31e"
Nov 04 00:00:57.389197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.389163    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:57.400956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.396053    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:57.458774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.458172    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:00:57.458774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.458231    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:00:57.747971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.747677    5074 generic.go:356] "Generic (PLEG): container finished" podID="fb75763c-f905-4f8b-9456-6a18f512f2a5" containerID="a1284f77b0b351d549fdaf3e253b2d244f6ecca98d6413068aeea655c6ab9f06" exitCode=0
Nov 04 00:00:57.747971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.747910    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" event={"ID":"fb75763c-f905-4f8b-9456-6a18f512f2a5","Type":"ContainerDied","Data":"a1284f77b0b351d549fdaf3e253b2d244f6ecca98d6413068aeea655c6ab9f06"}
Nov 04 00:00:57.762095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.762069    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 04 00:00:57.830796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.829323    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm\") pod \"fb75763c-f905-4f8b-9456-6a18f512f2a5\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") "
Nov 04 00:00:57.830796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.829460    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache\") pod \"fb75763c-f905-4f8b-9456-6a18f512f2a5\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") "
Nov 04 00:00:57.830796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.829543    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home\") pod \"fb75763c-f905-4f8b-9456-6a18f512f2a5\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") "
Nov 04 00:00:57.830796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.829678    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb\" (UniqueName: \"kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb\") pod \"fb75763c-f905-4f8b-9456-6a18f512f2a5\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") "
Nov 04 00:00:57.830796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.829778    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir\") pod \"fb75763c-f905-4f8b-9456-6a18f512f2a5\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") "
Nov 04 00:00:57.830796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.829838    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs\") pod \"fb75763c-f905-4f8b-9456-6a18f512f2a5\" (UID: \"fb75763c-f905-4f8b-9456-6a18f512f2a5\") "
Nov 04 00:00:57.844459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.844110    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache" pod "fb75763c-f905-4f8b-9456-6a18f512f2a5" (UID: "fb75763c-f905-4f8b-9456-6a18f512f2a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:57.850770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.848993    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home" pod "fb75763c-f905-4f8b-9456-6a18f512f2a5" (UID: "fb75763c-f905-4f8b-9456-6a18f512f2a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:57.850770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.849123    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir" pod "fb75763c-f905-4f8b-9456-6a18f512f2a5" (UID: "fb75763c-f905-4f8b-9456-6a18f512f2a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:57.864527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.864492    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs" pod "fb75763c-f905-4f8b-9456-6a18f512f2a5" (UID: "fb75763c-f905-4f8b-9456-6a18f512f2a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:57.867797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.866142    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb" pod "fb75763c-f905-4f8b-9456-6a18f512f2a5" (UID: "fb75763c-f905-4f8b-9456-6a18f512f2a5"). InnerVolumeSpecName "kube-api-access-dbkxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:57.867797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.866803    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm" pod "fb75763c-f905-4f8b-9456-6a18f512f2a5" (UID: "fb75763c-f905-4f8b-9456-6a18f512f2a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:57.944437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.944395    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:57.944437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.944438    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:57.944677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.944456    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:57.944677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.944472    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkxb\" (UniqueName: \"kubernetes.io/projected/fb75763c-f905-4f8b-9456-6a18f512f2a5-kube-api-access-dbkxb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:57.944677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.944487    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb75763c-f905-4f8b-9456-6a18f512f2a5-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:57.944677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:57.944501    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb75763c-f905-4f8b-9456-6a18f512f2a5-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:58.156463 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:00:58.150908    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:00:58.161949 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.160998    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"]
Nov 04 00:00:58.351498 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:00:58.351447    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fcde93_4243_4629_93cc_7a58fbe489ec.slice/crio-6d61202dfcc6e1e04951bdf5a6a22bef76c519fc32d807b51d0b16df838ee4e5 WatchSource:0}: Error finding container 6d61202dfcc6e1e04951bdf5a6a22bef76c519fc32d807b51d0b16df838ee4e5: Status 404 returned error can't find the container with id 6d61202dfcc6e1e04951bdf5a6a22bef76c519fc32d807b51d0b16df838ee4e5
Nov 04 00:00:58.374710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.372903    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"]
Nov 04 00:00:58.455763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.455503    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:00:58.455763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.455549    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:00:58.768724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.768677    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" event={"ID":"e90267b5-38ea-499b-aa99-2eb911054653","Type":"ContainerStarted","Data":"7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460"}
Nov 04 00:00:58.769284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.768769    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:00:58.769284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.768796    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" event={"ID":"e90267b5-38ea-499b-aa99-2eb911054653","Type":"ContainerStarted","Data":"57fe59cb41d4edae251f59474c534c07522f8976393389765794e909a4c357d6"}
Nov 04 00:00:58.770703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.770682    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"
Nov 04 00:00:58.770847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.770702    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6" event={"ID":"fb75763c-f905-4f8b-9456-6a18f512f2a5","Type":"ContainerDied","Data":"9fa8dcf08662ef5b6f0063eed007ce6acdf988f8a42fce7fc49e3c7c75451d10"}
Nov 04 00:00:58.770847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.770787    5074 scope.go:122] "RemoveContainer" containerID="a1284f77b0b351d549fdaf3e253b2d244f6ecca98d6413068aeea655c6ab9f06"
Nov 04 00:00:58.772811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.772789    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn" event={"ID":"17fcde93-4243-4629-93cc-7a58fbe489ec","Type":"ContainerStarted","Data":"aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d"}
Nov 04 00:00:58.772933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.772915    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn" event={"ID":"17fcde93-4243-4629-93cc-7a58fbe489ec","Type":"ContainerStarted","Data":"6d61202dfcc6e1e04951bdf5a6a22bef76c519fc32d807b51d0b16df838ee4e5"}
Nov 04 00:00:58.773401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.773384    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:00:58.797586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.797542    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" podStartSLOduration=2.797530976 podStartE2EDuration="2.797530976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:58.790424174 +0000 UTC m=+1923.890024320" watchObservedRunningTime="2026-11-04 00:00:58.797530976 +0000 UTC m=+1923.897131166"
Nov 04 00:00:58.860174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.860120    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn" podStartSLOduration=2.8601070010000003 podStartE2EDuration="2.860107001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:58.856459248 +0000 UTC m=+1923.956059393" watchObservedRunningTime="2026-11-04 00:00:58.860107001 +0000 UTC m=+1923.959707135"
Nov 04 00:00:58.875685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.875652    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"]
Nov 04 00:00:58.879760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:58.879718    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-330eb5/auth-test-model-330eb5-kserve-798c68db57-kp2d6"]
Nov 04 00:00:59.061552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:59.061477    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb75763c-f905-4f8b-9456-6a18f512f2a5" path="/var/lib/kubelet/pods/fb75763c-f905-4f8b-9456-6a18f512f2a5/volumes"
Nov 04 00:00:59.456152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:59.456118    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:00:59.456477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:00:59.456425    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:01:00.458760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:00.457514    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:01:00.458760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:00.457581    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:01:01.442588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:01.441877    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:01:01.456410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:01.455993    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:01:01.456410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:01.456049    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 04 00:01:01.625761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:01.625010    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:01:02.701062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:02.701023    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:01:02.702475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:02.702436    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:01:02.705663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:02.705638    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:01:05.196719 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:05.196643    5074 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-self-signed-certs: secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:01:05.197109 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:05.196731    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs podName:65b2acae-1a0f-4d9f-a643-08701153e9d9 nodeName:}" failed. No retries permitted until 2026-11-04 00:01:21.196710989 +0000 UTC m=+1946.296311136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs") pod "noten-model-ba68e7-kserve-68674f8d5c-sqwmj" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9") : secret "noten-model-ba68e7-kserve-self-signed-certs" not found
Nov 04 00:01:05.462805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.462702    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj" podUID="65b2acae-1a0f-4d9f-a643-08701153e9d9" containerName="main" containerID="cri-o://7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3" gracePeriod=45
Nov 04 00:01:05.571139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.571108    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:01:05.871361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.871335    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:01:05.908205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.908031    5074 generic.go:356] "Generic (PLEG): container finished" podID="65b2acae-1a0f-4d9f-a643-08701153e9d9" containerID="7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3" exitCode=0
Nov 04 00:01:05.908205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.908188    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj" event={"ID":"65b2acae-1a0f-4d9f-a643-08701153e9d9","Type":"ContainerDied","Data":"7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3"}
Nov 04 00:01:05.908478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.908228    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj" event={"ID":"65b2acae-1a0f-4d9f-a643-08701153e9d9","Type":"ContainerDied","Data":"6467c332610d37253eabad53bf999f9271fe134b2a75fca9100b8716f6095434"}
Nov 04 00:01:05.908478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.908250    5074 scope.go:122] "RemoveContainer" containerID="7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3"
Nov 04 00:01:05.908478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.908393    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"
Nov 04 00:01:05.921360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.919952    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home\") pod \"65b2acae-1a0f-4d9f-a643-08701153e9d9\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") "
Nov 04 00:01:05.921360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.920183    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache\") pod \"65b2acae-1a0f-4d9f-a643-08701153e9d9\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") "
Nov 04 00:01:05.921360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.920363    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx\" (UniqueName: \"kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx\") pod \"65b2acae-1a0f-4d9f-a643-08701153e9d9\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") "
Nov 04 00:01:05.921360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.920421    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs\") pod \"65b2acae-1a0f-4d9f-a643-08701153e9d9\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") "
Nov 04 00:01:05.921360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.920454    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm\") pod \"65b2acae-1a0f-4d9f-a643-08701153e9d9\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") "
Nov 04 00:01:05.921360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.920517    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir\") pod \"65b2acae-1a0f-4d9f-a643-08701153e9d9\" (UID: \"65b2acae-1a0f-4d9f-a643-08701153e9d9\") "
Nov 04 00:01:05.923486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.922509    5074 scope.go:122] "RemoveContainer" containerID="7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3"
Nov 04 00:01:05.923486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.923094    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache" pod "65b2acae-1a0f-4d9f-a643-08701153e9d9" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:05.923486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.923104    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home" pod "65b2acae-1a0f-4d9f-a643-08701153e9d9" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:05.923486 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:05.923196    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3\": container with ID starting with 7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3 not found: ID does not exist" containerID="7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3"
Nov 04 00:01:05.923486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.923228    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3"} err="failed to get container status \"7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3\": rpc error: code = NotFound desc = could not find container \"7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3\": container with ID starting with 7b318b7743a6b6c15d29ef0c2a11f68cf0a0278dcc4a65f34f0cfbf5251d96a3 not found: ID does not exist"
Nov 04 00:01:05.924160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.924138    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir" pod "65b2acae-1a0f-4d9f-a643-08701153e9d9" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:05.929612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.929587    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm" pod "65b2acae-1a0f-4d9f-a643-08701153e9d9" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:05.931126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.931100    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs" pod "65b2acae-1a0f-4d9f-a643-08701153e9d9" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:01:05.931578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:05.931544    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx" pod "65b2acae-1a0f-4d9f-a643-08701153e9d9" (UID: "65b2acae-1a0f-4d9f-a643-08701153e9d9"). InnerVolumeSpecName "kube-api-access-rttzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:06.022568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.022524    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:06.022568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.022567    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:06.022806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.022581    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:06.022806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.022600    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rttzx\" (UniqueName: \"kubernetes.io/projected/65b2acae-1a0f-4d9f-a643-08701153e9d9-kube-api-access-rttzx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:06.022806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.022616    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b2acae-1a0f-4d9f-a643-08701153e9d9-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:06.022806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.022629    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b2acae-1a0f-4d9f-a643-08701153e9d9-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:06.242225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.242188    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"]
Nov 04 00:01:06.246388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.246356    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ba68e7/noten-model-ba68e7-kserve-68674f8d5c-sqwmj"]
Nov 04 00:01:06.519573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.519487    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"]
Nov 04 00:01:06.520251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.520233    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fb75763c-f905-4f8b-9456-6a18f512f2a5" containerName="main"
Nov 04 00:01:06.520251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.520250    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb75763c-f905-4f8b-9456-6a18f512f2a5" containerName="main"
Nov 04 00:01:06.520466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.520286    5074 cpu_manager.go:414] "containerMap: removing container" podUID="65b2acae-1a0f-4d9f-a643-08701153e9d9" containerName="main"
Nov 04 00:01:06.520466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.520292    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b2acae-1a0f-4d9f-a643-08701153e9d9" containerName="main"
Nov 04 00:01:06.520466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.520409    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb75763c-f905-4f8b-9456-6a18f512f2a5" containerName="main"
Nov 04 00:01:06.520466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.520425    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b2acae-1a0f-4d9f-a643-08701153e9d9" containerName="main"
Nov 04 00:01:06.525896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.525870    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.528387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.528363    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"]
Nov 04 00:01:06.533140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.533117    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.534566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.534540    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8qq\" (UniqueName: \"kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq\") pod \"payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.534724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.534706    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.534812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.534779    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb95\" (UniqueName: \"kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95\") pod \"payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.534881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.534861    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.544149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.543647    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2697a0\""
Nov 04 00:01:06.553169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.552816    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"]
Nov 04 00:01:06.558544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.558517    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"]
Nov 04 00:01:06.578434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.578412    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:01:06.639790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.635964    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.639790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.636854    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb95\" (UniqueName: \"kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95\") pod \"payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.639790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.636902    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.639790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.637064    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8qq\" (UniqueName: \"kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq\") pod \"payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.639790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.638507    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.639790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.639082    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.663505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.663467    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8qq\" (UniqueName: \"kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq\") pod \"payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:06.667823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.667792    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb95\" (UniqueName: \"kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95\") pod \"payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.849816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.848920    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:06.863173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:06.863086    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:07.043842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.043798    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b2acae-1a0f-4d9f-a643-08701153e9d9" path="/var/lib/kubelet/pods/65b2acae-1a0f-4d9f-a643-08701153e9d9/volumes"
Nov 04 00:01:07.321999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.321969    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"]
Nov 04 00:01:07.327894 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:01:07.327859    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f550104_0924_4be3_b048_da09e218369b.slice/crio-97878703d2cd460d7c2715b8810ec3ea07d7905b7ab8742947a03c61efb40119 WatchSource:0}: Error finding container 97878703d2cd460d7c2715b8810ec3ea07d7905b7ab8742947a03c61efb40119: Status 404 returned error can't find the container with id 97878703d2cd460d7c2715b8810ec3ea07d7905b7ab8742947a03c61efb40119
Nov 04 00:01:07.329555 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:01:07.329513    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66eb257f_677e_4efa_be06_5466c93cc0cc.slice/crio-5c5afbcb2abef95c864d47ec7fd940348b8788b7da258fce6e160a73f9d667c0 WatchSource:0}: Error finding container 5c5afbcb2abef95c864d47ec7fd940348b8788b7da258fce6e160a73f9d667c0: Status 404 returned error can't find the container with id 5c5afbcb2abef95c864d47ec7fd940348b8788b7da258fce6e160a73f9d667c0
Nov 04 00:01:07.333259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.333234    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"]
Nov 04 00:01:07.851107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.851073    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"]
Nov 04 00:01:07.855921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.855899    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.867367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.867338    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"]
Nov 04 00:01:07.890136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.890094    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-58b75d-openshift-default-dockercfg-ks9wt\""
Nov 04 00:01:07.915370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915326    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915395    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915527    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915629    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915671    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915709    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915750    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwhv\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915804    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:07.915893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:07.915829    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.017507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.017465    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.017507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.017513    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.017757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.017539    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.017757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.017655    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwhv\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.017883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.017728    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.017883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.017796    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018032    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018129    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018168    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018229    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018347    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018525    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018662    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.018981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.018961    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.020933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.020906    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.021044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.020969    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.042357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.042320    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" event={"ID":"8f550104-0924-4be3-b048-da09e218369b","Type":"ContainerStarted","Data":"c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09"}
Nov 04 00:01:08.042525 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.042391    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" event={"ID":"8f550104-0924-4be3-b048-da09e218369b","Type":"ContainerStarted","Data":"97878703d2cd460d7c2715b8810ec3ea07d7905b7ab8742947a03c61efb40119"}
Nov 04 00:01:08.042525 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.042419    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:08.044802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.044719    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj" event={"ID":"66eb257f-677e-4efa-be06-5466c93cc0cc","Type":"ContainerStarted","Data":"e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25"}
Nov 04 00:01:08.045010 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.044995    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:08.045147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.045126    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj" event={"ID":"66eb257f-677e-4efa-be06-5466c93cc0cc","Type":"ContainerStarted","Data":"5c5afbcb2abef95c864d47ec7fd940348b8788b7da258fce6e160a73f9d667c0"}
Nov 04 00:01:08.046125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.046101    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.046485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.046462    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwhv\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.120755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.120612    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" podStartSLOduration=2.120589524 podStartE2EDuration="2.120589524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:08.113729886 +0000 UTC m=+1933.213330066" watchObservedRunningTime="2026-11-04 00:01:08.120589524 +0000 UTC m=+1933.220189670"
Nov 04 00:01:08.150191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.150125    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj" podStartSLOduration=2.150102947 podStartE2EDuration="2.150102947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:08.147000174 +0000 UTC m=+1933.246600318" watchObservedRunningTime="2026-11-04 00:01:08.150102947 +0000 UTC m=+1933.249703092"
Nov 04 00:01:08.177085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.177045    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:08.496205 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:01:08.496171    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb0e01c_2c82_419d_bc2d_d1d3477e607d.slice/crio-e59f1deb3f461998aa257b556b65e39c46d54fa39d8933799a325e6822eabbd7 WatchSource:0}: Error finding container e59f1deb3f461998aa257b556b65e39c46d54fa39d8933799a325e6822eabbd7: Status 404 returned error can't find the container with id e59f1deb3f461998aa257b556b65e39c46d54fa39d8933799a325e6822eabbd7
Nov 04 00:01:08.497556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.497531    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"]
Nov 04 00:01:08.499441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.499411    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:08.499520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.499473    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:08.499520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:08.499503    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:09.055374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.055335    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" event={"ID":"1bb0e01c-2c82-419d-bc2d-d1d3477e607d","Type":"ContainerStarted","Data":"8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1"}
Nov 04 00:01:09.055552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.055397    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" event={"ID":"1bb0e01c-2c82-419d-bc2d-d1d3477e607d","Type":"ContainerStarted","Data":"e59f1deb3f461998aa257b556b65e39c46d54fa39d8933799a325e6822eabbd7"}
Nov 04 00:01:09.100136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.100078    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podStartSLOduration=2.100061634 podStartE2EDuration="2.100061634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:09.089945697 +0000 UTC m=+1934.189545843" watchObservedRunningTime="2026-11-04 00:01:09.100061634 +0000 UTC m=+1934.199661810"
Nov 04 00:01:09.181842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.181806    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:09.182487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.182459    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:09.182549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.182521    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:09.789408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:09.789376    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:01:10.177687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:10.177608    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:10.177687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:10.177661    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:10.897006 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:10.896969    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:01:11.177527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:11.177437    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:11.177837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:11.177802    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:12.181762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:12.180138    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:12.181762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:12.180206    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:13.178349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:13.178311    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:13.178562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:13.178372    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:14.178314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:14.178276    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:14.178884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:14.178331    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:15.178277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:15.178237    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 04 00:01:15.178485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:15.178300    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 04 00:01:16.185378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:16.185342    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:16.186546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:16.186521    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:16.187346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:16.187323    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:01:16.477708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:16.473430    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"]
Nov 04 00:01:16.477708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:16.473690    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy" containerID="cri-o://4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218" gracePeriod=30
Nov 04 00:01:17.685892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.685858    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"]
Nov 04 00:01:17.690090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.690071    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.703413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.703385    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9a5948-openshift-default-dockercfg-28jns\""
Nov 04 00:01:17.713749 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.713705    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"]
Nov 04 00:01:17.774393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.774352    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"]
Nov 04 00:01:17.774597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.774573    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy" containerID="cri-o://ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6" gracePeriod=30
Nov 04 00:01:17.839311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839269    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839369    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwsr\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839443    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839482    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839541    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839594    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839814    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.839956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.839934    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.840067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.840043    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941158    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941230    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941262    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwsr\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941301    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941332    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941357    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941398    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941563    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941696    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941713    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.941809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.941732    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.942026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.942004    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.942350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.942322    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.942490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.942380    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.944012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.943990    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.944106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.944032    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.955056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.955033    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:17.955257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:17.955238    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwsr\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:18.009393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:18.009352    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:18.387977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:18.387860    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"]
Nov 04 00:01:18.458088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:18.458037    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:18.458273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:18.458254    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:18.458429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:18.458409    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:19.058619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.058587    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:01:19.059246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.059046    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:01:19.319387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.319291    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" event={"ID":"81abacba-142f-4373-b6db-62e0f6e5bd5f","Type":"ContainerStarted","Data":"4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396"}
Nov 04 00:01:19.319688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.319668    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" event={"ID":"81abacba-142f-4373-b6db-62e0f6e5bd5f","Type":"ContainerStarted","Data":"862bc81ab06d2d447aa96bbd3d3c382d86fbc8681217f72cb7e056a729bd3368"}
Nov 04 00:01:19.356635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.356578    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podStartSLOduration=2.356564863 podStartE2EDuration="2.356564863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:19.352426092 +0000 UTC m=+1944.452026237" watchObservedRunningTime="2026-11-04 00:01:19.356564863 +0000 UTC m=+1944.456165006"
Nov 04 00:01:19.643064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.642009    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"]
Nov 04 00:01:19.643064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:19.642264    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy" containerID="cri-o://ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa" gracePeriod=30
Nov 04 00:01:20.031480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:20.030819    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:20.031480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:20.030946    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 04 00:01:20.031480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:20.030989    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 04 00:01:21.033921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:21.033868    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 04 00:01:21.034494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:21.033934    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 04 00:01:21.433850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:21.433394    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"]
Nov 04 00:01:21.433850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:21.433610    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy" containerID="cri-o://47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8" gracePeriod=30
Nov 04 00:01:21.876239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:21.876207    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 04 00:01:22.023822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.023239    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 04 00:01:22.023822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.023304    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.069787    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.070998    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.071096    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.071147    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.071222    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.071296    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.072667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.071355    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.073587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.073326    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:22.073587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.073390    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.073587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.073508    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo\") pod \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\" (UID: \"823da97b-dcfd-4dbd-94fe-e6af3dbac78d\") "
Nov 04 00:01:22.075346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.074218    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:22.075346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.074523    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.075346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.074542    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.075346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.074879    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:22.075346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.075026    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:22.075346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.075285    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:22.075919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.075466    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:22.076643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.076619    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "kube-api-access-w5qvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:22.076912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.076883    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:22.076984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.076916    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy" pod "823da97b-dcfd-4dbd-94fe-e6af3dbac78d" (UID: "823da97b-dcfd-4dbd-94fe-e6af3dbac78d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:22.176099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176066    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.176099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176096    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5qvj\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-kube-api-access-w5qvj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.176099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176107    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.176405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176115    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.176405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176124    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.176405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176132    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.176405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.176161    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823da97b-dcfd-4dbd-94fe-e6af3dbac78d-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:22.355082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.355000    5074 generic.go:356] "Generic (PLEG): container finished" podID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerID="4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218" exitCode=0
Nov 04 00:01:22.355506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.355322    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"
Nov 04 00:01:22.355792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.355356    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" event={"ID":"823da97b-dcfd-4dbd-94fe-e6af3dbac78d","Type":"ContainerDied","Data":"4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218"}
Nov 04 00:01:22.355878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.355859    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j" event={"ID":"823da97b-dcfd-4dbd-94fe-e6af3dbac78d","Type":"ContainerDied","Data":"68d3b3738b547bc7a0c31f0f38a85cd81ba3b2037fcde17dfda7bed041b5964f"}
Nov 04 00:01:22.355933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.355893    5074 scope.go:122] "RemoveContainer" containerID="4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218"
Nov 04 00:01:22.384387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.384119    5074 scope.go:122] "RemoveContainer" containerID="4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218"
Nov 04 00:01:22.384516 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:22.384441    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218\": container with ID starting with 4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218 not found: ID does not exist" containerID="4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218"
Nov 04 00:01:22.384516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.384477    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218"} err="failed to get container status \"4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218\": rpc error: code = NotFound desc = could not find container \"4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218\": container with ID starting with 4ef347a38d385fbaa5bd4099fd9f11f07ac7ae1ee568e7f4a9fb4f36db7ec218 not found: ID does not exist"
Nov 04 00:01:22.395222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.395088    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"]
Nov 04 00:01:22.410681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:22.410648    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-x989j"]
Nov 04 00:01:23.112273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.105655    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 04 00:01:23.112273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.105717    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 04 00:01:23.122762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.121288    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" path="/var/lib/kubelet/pods/823da97b-dcfd-4dbd-94fe-e6af3dbac78d/volumes"
Nov 04 00:01:23.162550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.161013    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"]
Nov 04 00:01:23.162550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.162324    5074 cpu_manager.go:414] "containerMap: removing container" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy"
Nov 04 00:01:23.162550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.162346    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy"
Nov 04 00:01:23.162796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.162604    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="823da97b-dcfd-4dbd-94fe-e6af3dbac78d" containerName="istio-proxy"
Nov 04 00:01:23.167512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.167489    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.177157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.177097    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2341ca-openshift-default-dockercfg-s4vcn\""
Nov 04 00:01:23.192909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.192876    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"]
Nov 04 00:01:23.269411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.269377    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 04 00:01:23.297342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.297277    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.297604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.297567    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.329831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.297710    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.329831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.302835    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:23.329831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.303103    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.345962    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346016    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346106    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346183    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346235    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346285    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token\") pod \"347a2c82-670c-455f-b442-482ccb9e7fa1\" (UID: \"347a2c82-670c-455f-b442-482ccb9e7fa1\") "
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346563    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346637    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.346851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346791    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346873    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346941    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.346978    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.347041    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.347075    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647hf\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.347302    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.347732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.347517    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.347732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.347546    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.348869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.348076    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:23.348869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.348299    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:23.349413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.349386    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:23.349641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.349614    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:23.352476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.352317    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "kube-api-access-hstlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:23.352856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.352678    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:23.353055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.353020    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token" pod "347a2c82-670c-455f-b442-482ccb9e7fa1" (UID: "347a2c82-670c-455f-b442-482ccb9e7fa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:23.370883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.370850    5074 generic.go:356] "Generic (PLEG): container finished" podID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerID="ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6" exitCode=0
Nov 04 00:01:23.370992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.370973    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" event={"ID":"347a2c82-670c-455f-b442-482ccb9e7fa1","Type":"ContainerDied","Data":"ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6"}
Nov 04 00:01:23.371049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.371028    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf" event={"ID":"347a2c82-670c-455f-b442-482ccb9e7fa1","Type":"ContainerDied","Data":"afacac5a549750daf027f13d1bc78dfd23e5799abdab8bca8f5b5b6dc84f11b3"}
Nov 04 00:01:23.371096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.371058    5074 scope.go:122] "RemoveContainer" containerID="ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6"
Nov 04 00:01:23.371197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.371183    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"
Nov 04 00:01:23.387489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.387462    5074 scope.go:122] "RemoveContainer" containerID="ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6"
Nov 04 00:01:23.387943 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:23.387911    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6\": container with ID starting with ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6 not found: ID does not exist" containerID="ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6"
Nov 04 00:01:23.388052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.387957    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6"} err="failed to get container status \"ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6\": rpc error: code = NotFound desc = could not find container \"ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6\": container with ID starting with ef5768315efaca56ad8dde2b95e4b226e32e0736d753d9fc2ab4094b0b6a84f6 not found: ID does not exist"
Nov 04 00:01:23.455764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455521    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.455764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455647    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.455764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455690    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455786    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455854    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455905    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455934    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.455985    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456017    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647hf\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456108    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456128    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstlh\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-kube-api-access-hstlh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456142    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456158    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347a2c82-670c-455f-b442-482ccb9e7fa1-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456172    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456188    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.456379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.456202    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347a2c82-670c-455f-b442-482ccb9e7fa1-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:23.459429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.458376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.459429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.458684    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.459429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.459121    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.459696 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.459447    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.459775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.459692    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.462579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.462525    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.463568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.463546    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.469340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.469317    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"]
Nov 04 00:01:23.469938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.469923    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647hf\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.471491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.471459    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.477116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.477090    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-xm5zf"]
Nov 04 00:01:23.568214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.568184    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:23.940905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.934315    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"]
Nov 04 00:01:23.943096 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:01:23.943056    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2ed446_5d52_4b60_8125_8a28850bb5c5.slice/crio-e2f99028d99e57b101ecfe9fef87f8e11df38f3446ec0c16fcb2b9b7e6e598b2 WatchSource:0}: Error finding container e2f99028d99e57b101ecfe9fef87f8e11df38f3446ec0c16fcb2b9b7e6e598b2: Status 404 returned error can't find the container with id e2f99028d99e57b101ecfe9fef87f8e11df38f3446ec0c16fcb2b9b7e6e598b2
Nov 04 00:01:23.947496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.947416    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:23.947684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.947641    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:23.947893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:23.947826    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:01:24.018773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:24.017889    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 04 00:01:24.018773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:24.018405    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 04 00:01:24.381043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:24.380992    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" event={"ID":"fd2ed446-5d52-4b60-8125-8a28850bb5c5","Type":"ContainerStarted","Data":"ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5"}
Nov 04 00:01:24.381043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:24.381049    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" event={"ID":"fd2ed446-5d52-4b60-8125-8a28850bb5c5","Type":"ContainerStarted","Data":"e2f99028d99e57b101ecfe9fef87f8e11df38f3446ec0c16fcb2b9b7e6e598b2"}
Nov 04 00:01:24.568724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:24.568687    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:24.980141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:24.980107    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 04 00:01:25.026761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.010911    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 04 00:01:25.026761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.010990    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 04 00:01:25.026761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.021961    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podStartSLOduration=2.02194281 podStartE2EDuration="2.02194281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:24.430259643 +0000 UTC m=+1949.529859788" watchObservedRunningTime="2026-11-04 00:01:25.02194281 +0000 UTC m=+1950.121542954"
Nov 04 00:01:25.058560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.058521    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" path="/var/lib/kubelet/pods/347a2c82-670c-455f-b442-482ccb9e7fa1/volumes"
Nov 04 00:01:25.083354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083317    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.083354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083365    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.083617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083596    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.083708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083684    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.083831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083783    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.083831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083817    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:25.083947 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083880    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:25.083947 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083879    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.084064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083951    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.084064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.083998    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.084176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.084096    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:25.084397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.084370    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:25.084491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.084440    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket\") pod \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\" (UID: \"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2\") "
Nov 04 00:01:25.084698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.084677    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:25.085291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.085264    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.085404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.085297    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.085404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.085314    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.085404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.085328    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.085404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.085342    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.086422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.086393    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "kube-api-access-56xw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:25.087834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.087798    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:25.087900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.087805    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:25.090131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.090105    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo" pod "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" (UID: "aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:25.186583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.186544    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.186583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.186582    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xw8\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-kube-api-access-56xw8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.186583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.186594    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.186817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.186603    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:25.418361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.418245    5074 generic.go:356] "Generic (PLEG): container finished" podID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerID="ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa" exitCode=0
Nov 04 00:01:25.422849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.420275    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"
Nov 04 00:01:25.423603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.423569    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" event={"ID":"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2","Type":"ContainerDied","Data":"ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa"}
Nov 04 00:01:25.423841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.423814    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8" event={"ID":"aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2","Type":"ContainerDied","Data":"dd2390dbda662dfc462221ebb324bf19c6c0ee3208ce8b6e55ac9b897b6e521b"}
Nov 04 00:01:25.423976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.423962    5074 scope.go:122] "RemoveContainer" containerID="ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa"
Nov 04 00:01:25.464163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.464141    5074 scope.go:122] "RemoveContainer" containerID="ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa"
Nov 04 00:01:25.464553 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:25.464512    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa\": container with ID starting with ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa not found: ID does not exist" containerID="ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa"
Nov 04 00:01:25.464694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.464555    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa"} err="failed to get container status \"ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa\": rpc error: code = NotFound desc = could not find container \"ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa\": container with ID starting with ef7740efdeceb0167324e5d63ed26618f37f1d0b7a6df3b5813148b8fc14abaa not found: ID does not exist"
Nov 04 00:01:25.480235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.480205    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"]
Nov 04 00:01:25.484209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.484184    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-767g8"]
Nov 04 00:01:25.575800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.571066    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:01:25.575800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:25.571139    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 04 00:01:26.045424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.043610    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:26.045424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.045255    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:26.046623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.046603    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:01:26.569735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.569692    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:01:26.570107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.569836    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 04 00:01:26.841256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.841224    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 04 00:01:26.887236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887195    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887292    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887327    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887359    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887386    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887440    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887491    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887564    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.887668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.887657    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token\") pod \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\" (UID: \"e46008f3-d315-42f4-b9fc-79acf87fb9cd\") "
Nov 04 00:01:26.888159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.888131    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:26.889227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.888857    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.889227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.889015    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.889227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.889074    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.889227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.889082    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.889227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.889096    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.889227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.889111    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.891485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.891458    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.891790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.891732    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "kube-api-access-pp2kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:26.931960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.923905    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:26.958761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.958022    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo" pod "e46008f3-d315-42f4-b9fc-79acf87fb9cd" (UID: "e46008f3-d315-42f4-b9fc-79acf87fb9cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:26.991072 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991027    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.991310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991276    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.991417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991404    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.991500 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991488    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2kb\" (UniqueName: \"kubernetes.io/projected/e46008f3-d315-42f4-b9fc-79acf87fb9cd-kube-api-access-pp2kb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.991576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991566    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.991733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991719    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.991864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:26.991849    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46008f3-d315-42f4-b9fc-79acf87fb9cd-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:27.057799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.057591    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" path="/var/lib/kubelet/pods/aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2/volumes"
Nov 04 00:01:27.496764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.484621    5074 generic.go:356] "Generic (PLEG): container finished" podID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerID="47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8" exitCode=0
Nov 04 00:01:27.496764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.485958    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"
Nov 04 00:01:27.496764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.489801    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" event={"ID":"e46008f3-d315-42f4-b9fc-79acf87fb9cd","Type":"ContainerDied","Data":"47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8"}
Nov 04 00:01:27.496764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.489865    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm" event={"ID":"e46008f3-d315-42f4-b9fc-79acf87fb9cd","Type":"ContainerDied","Data":"98b797c9d1d2b0ab6411c5a2eb5e40ec1285c9981bc1b8377ff9b1763f48062c"}
Nov 04 00:01:27.496764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.490293    5074 scope.go:122] "RemoveContainer" containerID="47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8"
Nov 04 00:01:27.576764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.569089    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:01:27.576764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.569179    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 04 00:01:27.617767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.611963    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"]
Nov 04 00:01:27.656662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.654385    5074 scope.go:122] "RemoveContainer" containerID="47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8"
Nov 04 00:01:27.659756 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:27.659666    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8\": container with ID starting with 47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8 not found: ID does not exist" containerID="47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8"
Nov 04 00:01:27.659756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.659714    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8"} err="failed to get container status \"47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8\": rpc error: code = NotFound desc = could not find container \"47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8\": container with ID starting with 47ae49ade0df7666cc0fd7905a619b86b4b692164414e4223eb38d3b3a5bdaa8 not found: ID does not exist"
Nov 04 00:01:27.675116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:27.674715    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vgxkm"]
Nov 04 00:01:28.173448 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:28.173374    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:01:28.569343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:28.569306    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:01:28.569636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:28.569607    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 04 00:01:29.036190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:29.036145    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" path="/var/lib/kubelet/pods/e46008f3-d315-42f4-b9fc-79acf87fb9cd/volumes"
Nov 04 00:01:29.568961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:29.568907    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:01:29.569184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:29.568986    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 04 00:01:30.569931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:30.569841    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:01:30.570962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:30.569940    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 04 00:01:31.612287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:31.612250    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:31.613524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:31.613388    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:31.614790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:31.614769    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:01:48.963340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.963303    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"]
Nov 04 00:01:48.964089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964074    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy"
Nov 04 00:01:48.964152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964090    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy"
Nov 04 00:01:48.964152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964115    5074 cpu_manager.go:414] "containerMap: removing container" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy"
Nov 04 00:01:48.964152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964121    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy"
Nov 04 00:01:48.964152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964134    5074 cpu_manager.go:414] "containerMap: removing container" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy"
Nov 04 00:01:48.964152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964139    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy"
Nov 04 00:01:48.964332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964251    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46008f3-d315-42f4-b9fc-79acf87fb9cd" containerName="istio-proxy"
Nov 04 00:01:48.964332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964263    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa62ff0f-0bdf-4d23-82a9-7ff8b04075c2" containerName="istio-proxy"
Nov 04 00:01:48.964332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.964271    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="347a2c82-670c-455f-b442-482ccb9e7fa1" containerName="istio-proxy"
Nov 04 00:01:48.968168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.968150    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:48.976688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.976666    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1b5fca-dockercfg-4nlf7\""
Nov 04 00:01:48.978122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.978097    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1b5fca\""
Nov 04 00:01:48.979996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:48.978561    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"]
Nov 04 00:01:49.007672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.007635    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"]
Nov 04 00:01:49.021224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.021188    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"]
Nov 04 00:01:49.021381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.021333    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.068085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.066889    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.068680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.068614    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2hs\" (UniqueName: \"kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs\") pod \"payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.194476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.194423    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2hs\" (UniqueName: \"kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs\") pod \"payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.221484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.194857    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47c42\" (UniqueName: \"kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42\") pod \"payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.221816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.221789    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.222156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.222131    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.240913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.240871    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.310811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.310781    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2hs\" (UniqueName: \"kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs\") pod \"payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.324958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.324236    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.324958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.324414    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47c42\" (UniqueName: \"kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42\") pod \"payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.325529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.325504    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.348609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.348580    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47c42\" (UniqueName: \"kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42\") pod \"payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.368225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.368197    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:49.587731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.587645    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:49.681138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.680812    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"]
Nov 04 00:01:49.891477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.891330    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"]
Nov 04 00:01:49.937762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.932335    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" event={"ID":"430961a4-6dcd-4bff-be54-addc4e3cade4","Type":"ContainerStarted","Data":"71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920"}
Nov 04 00:01:49.937762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.932409    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" event={"ID":"430961a4-6dcd-4bff-be54-addc4e3cade4","Type":"ContainerStarted","Data":"561ad9b866a8dcedc5e67998aa49df9e37ac014d00a83d00b3dd281a0332ea9c"}
Nov 04 00:01:49.937762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:49.934317    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:01:50.020774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:50.016672    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" podStartSLOduration=2.016648657 podStartE2EDuration="2.016648657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:49.977142487 +0000 UTC m=+1975.076742632" watchObservedRunningTime="2026-11-04 00:01:50.016648657 +0000 UTC m=+1975.116248804"
Nov 04 00:01:50.946029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:50.945993    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s" event={"ID":"5ed92c25-9518-438e-b30a-175d7aa94e3c","Type":"ContainerStarted","Data":"e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767"}
Nov 04 00:01:50.946251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:50.946054    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s" event={"ID":"5ed92c25-9518-438e-b30a-175d7aa94e3c","Type":"ContainerStarted","Data":"e72d230deaaf23c84ea60faa6f178d759a08dbc10e4a94129409a2b1bd7d91cc"}
Nov 04 00:01:50.946251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:50.946079    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:01:50.966540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:50.966492    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s" podStartSLOduration=2.966480181 podStartE2EDuration="2.966480181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:50.965342548 +0000 UTC m=+1976.064942695" watchObservedRunningTime="2026-11-04 00:01:50.966480181 +0000 UTC m=+1976.066080360"
Nov 04 00:01:51.671001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.670964    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"]
Nov 04 00:01:51.676714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.676688    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:51.683657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.683628    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8895cb\""
Nov 04 00:01:51.687911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.687885    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"]
Nov 04 00:01:51.701932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.701890    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:51.702302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.702276    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95d6t\" (UniqueName: \"kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t\") pod \"payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:51.727179 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.727146    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"]
Nov 04 00:01:51.736352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.736322    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:51.742752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.742701    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"]
Nov 04 00:01:51.785620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.760147    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8895cb-dockercfg-jtg49\""
Nov 04 00:01:51.816212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.816167    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:51.816461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.816394    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95d6t\" (UniqueName: \"kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t\") pod \"payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:51.817780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:51.817733    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:52.041786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.041733    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cls8\" (UniqueName: \"kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8\") pod \"payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.042449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.042424    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.052516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.052481    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95d6t\" (UniqueName: \"kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t\") pod \"payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:52.144864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.144826    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cls8\" (UniqueName: \"kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8\") pod \"payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.145058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.144953    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.146018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.145991    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.168668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.168639    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cls8\" (UniqueName: \"kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8\") pod \"payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.305110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.304999    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:52.341066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.341028    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:52.689171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.689131    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"]
Nov 04 00:01:52.693631 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:01:52.693604    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbde246c_8dc0_455f_bcf9_946d19291496.slice/crio-e47b46ef0a364355bd4476de518740bdc479b164e6208af58d0a5f30d4e51455 WatchSource:0}: Error finding container e47b46ef0a364355bd4476de518740bdc479b164e6208af58d0a5f30d4e51455: Status 404 returned error can't find the container with id e47b46ef0a364355bd4476de518740bdc479b164e6208af58d0a5f30d4e51455
Nov 04 00:01:52.735355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:52.735329    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"]
Nov 04 00:01:53.041113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.041025    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd" event={"ID":"dbde246c-8dc0-455f-bcf9-946d19291496","Type":"ContainerStarted","Data":"79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83"}
Nov 04 00:01:53.041113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.041094    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd" event={"ID":"dbde246c-8dc0-455f-bcf9-946d19291496","Type":"ContainerStarted","Data":"e47b46ef0a364355bd4476de518740bdc479b164e6208af58d0a5f30d4e51455"}
Nov 04 00:01:53.041353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.041247    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:01:53.043422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.043396    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" event={"ID":"c633171b-c6c0-4f2f-9723-b8aa527533df","Type":"ContainerStarted","Data":"6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8"}
Nov 04 00:01:53.043565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.043436    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" event={"ID":"c633171b-c6c0-4f2f-9723-b8aa527533df","Type":"ContainerStarted","Data":"819759fe75810223edde88860125788475f69496ba02dfa89f1d018a2bacfc2f"}
Nov 04 00:01:53.043565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.043533    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:01:53.069147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.069087    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd" podStartSLOduration=2.069070003 podStartE2EDuration="2.069070003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:53.066682641 +0000 UTC m=+1978.166282786" watchObservedRunningTime="2026-11-04 00:01:53.069070003 +0000 UTC m=+1978.168670147"
Nov 04 00:01:53.090510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:53.090455    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" podStartSLOduration=2.090441166 podStartE2EDuration="2.090441166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:53.088588001 +0000 UTC m=+1978.188188144" watchObservedRunningTime="2026-11-04 00:01:53.090441166 +0000 UTC m=+1978.190041309"
Nov 04 00:01:58.025443 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:01:58.025398    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:01:58.263301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:58.263262    5074 scope.go:122] "RemoveContainer" containerID="08f4eed46c99673b4a002653aea7c3b4cb5966f5b63e26ebfd609a4f9a746f86"
Nov 04 00:01:58.278869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:01:58.278838    5074 scope.go:122] "RemoveContainer" containerID="5c1ade97bef61d4d317bf230550209dc2ec1c3ab2160d5f4ebb77d5b4cef9fd8"
Nov 04 00:02:02.036034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:02.036001    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:02:02.036658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:02.036068    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:02:04.222295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:04.222257    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:02:04.224390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:04.224364    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:02:28.031183 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:02:28.031141    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:02:40.278699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:40.278618    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"]
Nov 04 00:02:40.288049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:40.288020    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"]
Nov 04 00:02:45.279399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.279355    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" podUID="424cc865-b851-4e2b-ab02-64387e84b727" containerName="payload-processing" containerID="cri-o://b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d" gracePeriod=25
Nov 04 00:02:45.288594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.288558    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b" podUID="a87e1087-c0d8-43a8-8566-6584e5c3c1c7" containerName="payload-pre-processing" containerID="cri-o://c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda" gracePeriod=25
Nov 04 00:02:45.618389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.618357    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:02:45.650841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.650810    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume\") pod \"424cc865-b851-4e2b-ab02-64387e84b727\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") "
Nov 04 00:02:45.651022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.650946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt\" (UniqueName: \"kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt\") pod \"424cc865-b851-4e2b-ab02-64387e84b727\" (UID: \"424cc865-b851-4e2b-ab02-64387e84b727\") "
Nov 04 00:02:45.651223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.651199    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume" pod "424cc865-b851-4e2b-ab02-64387e84b727" (UID: "424cc865-b851-4e2b-ab02-64387e84b727"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:02:45.651464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.651444    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc865-b851-4e2b-ab02-64387e84b727-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.653109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.653086    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt" pod "424cc865-b851-4e2b-ab02-64387e84b727" (UID: "424cc865-b851-4e2b-ab02-64387e84b727"). InnerVolumeSpecName "kube-api-access-xjzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:02:45.666777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.666710    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:02:45.752381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.752340    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b\" (UniqueName: \"kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b\") pod \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") "
Nov 04 00:02:45.752591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.752476    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume\") pod \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\" (UID: \"a87e1087-c0d8-43a8-8566-6584e5c3c1c7\") "
Nov 04 00:02:45.752988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.752957    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume" pod "a87e1087-c0d8-43a8-8566-6584e5c3c1c7" (UID: "a87e1087-c0d8-43a8-8566-6584e5c3c1c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:02:45.753198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.753166    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjzvt\" (UniqueName: \"kubernetes.io/projected/424cc865-b851-4e2b-ab02-64387e84b727-kube-api-access-xjzvt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.753198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.753195    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.754857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.754817    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b" pod "a87e1087-c0d8-43a8-8566-6584e5c3c1c7" (UID: "a87e1087-c0d8-43a8-8566-6584e5c3c1c7"). InnerVolumeSpecName "kube-api-access-9995b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:02:45.855004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.854910    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9995b\" (UniqueName: \"kubernetes.io/projected/a87e1087-c0d8-43a8-8566-6584e5c3c1c7-kube-api-access-9995b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.886896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.886862    5074 generic.go:356] "Generic (PLEG): container finished" podID="a87e1087-c0d8-43a8-8566-6584e5c3c1c7" containerID="c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda" exitCode=0
Nov 04 00:02:45.887106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.886939    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"
Nov 04 00:02:45.887106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.886954    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b" event={"ID":"a87e1087-c0d8-43a8-8566-6584e5c3c1c7","Type":"ContainerDied","Data":"c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda"}
Nov 04 00:02:45.887106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.887021    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b" event={"ID":"a87e1087-c0d8-43a8-8566-6584e5c3c1c7","Type":"ContainerDied","Data":"4c35caa966f81abed43b49310765fb466321bb953b83cb636715106180fc2a5f"}
Nov 04 00:02:45.887106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.887060    5074 scope.go:122] "RemoveContainer" containerID="c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda"
Nov 04 00:02:45.888991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.888968    5074 generic.go:356] "Generic (PLEG): container finished" podID="424cc865-b851-4e2b-ab02-64387e84b727" containerID="b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d" exitCode=0
Nov 04 00:02:45.889116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.889027    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"
Nov 04 00:02:45.889116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.889056    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" event={"ID":"424cc865-b851-4e2b-ab02-64387e84b727","Type":"ContainerDied","Data":"b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d"}
Nov 04 00:02:45.889116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.889105    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" event={"ID":"424cc865-b851-4e2b-ab02-64387e84b727","Type":"ContainerDied","Data":"de88696c3b8f15a6f88290f45d7585e3ec458ae4de676bb5cefc7c0e97d14ba2"}
Nov 04 00:02:45.899949 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.899931    5074 scope.go:122] "RemoveContainer" containerID="c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda"
Nov 04 00:02:45.900264 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:02:45.900239    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda\": container with ID starting with c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda not found: ID does not exist" containerID="c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda"
Nov 04 00:02:45.900355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.900275    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda"} err="failed to get container status \"c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda\": rpc error: code = NotFound desc = could not find container \"c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda\": container with ID starting with c0e9418199418a8abf9ebf76fae06cef7e063fff68d23f5335ef8afbba906eda not found: ID does not exist"
Nov 04 00:02:45.900355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.900301    5074 scope.go:122] "RemoveContainer" containerID="b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d"
Nov 04 00:02:45.912464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.912441    5074 scope.go:122] "RemoveContainer" containerID="b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d"
Nov 04 00:02:45.912818 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:02:45.912795    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d\": container with ID starting with b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d not found: ID does not exist" containerID="b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d"
Nov 04 00:02:45.912895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.912828    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d"} err="failed to get container status \"b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d\": rpc error: code = NotFound desc = could not find container \"b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d\": container with ID starting with b426a1ba5b83776626ec25a22ee5ee9c4446192c3f09e607ab5cd73c626bd59d not found: ID does not exist"
Nov 04 00:02:45.936444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.936407    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"]
Nov 04 00:02:45.949311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.949279    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h"]
Nov 04 00:02:45.952893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.952862    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"]
Nov 04 00:02:45.957420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:45.957393    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27f8a1-74c8947b8c-w9j2b"]
Nov 04 00:02:46.568666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:46.568561    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-27f8a1-5c47c4df6b-jrm5h" podUID="424cc865-b851-4e2b-ab02-64387e84b727" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.226:50052\" within 1s: context deadline exceeded"
Nov 04 00:02:46.568666 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:02:46.568639    5074 logging.go:55] [core] [Channel #3835 SubChannel #3836]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.226:50052", ServerName: "10.132.0.226:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.226:50052: operation was canceled"
Nov 04 00:02:47.025215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:47.025179    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424cc865-b851-4e2b-ab02-64387e84b727" path="/var/lib/kubelet/pods/424cc865-b851-4e2b-ab02-64387e84b727/volumes"
Nov 04 00:02:47.025661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:47.025646    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a87e1087-c0d8-43a8-8566-6584e5c3c1c7" path="/var/lib/kubelet/pods/a87e1087-c0d8-43a8-8566-6584e5c3c1c7/volumes"
Nov 04 00:02:57.346575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.346536    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"]
Nov 04 00:02:57.347408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.347388    5074 cpu_manager.go:414] "containerMap: removing container" podUID="424cc865-b851-4e2b-ab02-64387e84b727" containerName="payload-processing"
Nov 04 00:02:57.347455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.347410    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424cc865-b851-4e2b-ab02-64387e84b727" containerName="payload-processing"
Nov 04 00:02:57.347455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.347441    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a87e1087-c0d8-43a8-8566-6584e5c3c1c7" containerName="payload-pre-processing"
Nov 04 00:02:57.347455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.347446    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87e1087-c0d8-43a8-8566-6584e5c3c1c7" containerName="payload-pre-processing"
Nov 04 00:02:57.347576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.347567    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87e1087-c0d8-43a8-8566-6584e5c3c1c7" containerName="payload-pre-processing"
Nov 04 00:02:57.347612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.347585    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="424cc865-b851-4e2b-ab02-64387e84b727" containerName="payload-processing"
Nov 04 00:02:57.351687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.351658    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.361378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.361345    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"]
Nov 04 00:02:57.371950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.371914    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-86deca\""
Nov 04 00:02:57.413848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.413803    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.414078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.413883    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfpr\" (UniqueName: \"kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr\") pod \"payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.444356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.444325    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"]
Nov 04 00:02:57.457326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.457291    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"]
Nov 04 00:02:57.457626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.457439    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.495994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.488363    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-86deca-dockercfg-qnsqc\""
Nov 04 00:02:57.519893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.519853    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.520177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.520151    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfpr\" (UniqueName: \"kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr\") pod \"payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.520341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.520323    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.520494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.520477    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8t22\" (UniqueName: \"kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22\") pod \"payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.521713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.521688    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.657822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.642755    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.678761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.676138    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8t22\" (UniqueName: \"kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22\") pod \"payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.678761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.659139    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.679916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.679855    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfpr\" (UniqueName: \"kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr\") pod \"payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:57.694235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.694202    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8t22\" (UniqueName: \"kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22\") pod \"payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.798827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.798791    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:57.973603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:57.973564    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:58.052569 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:02:58.048968    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:02:58.102509 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:02:58.102477    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad917029_10ba_40c9_9b87_988c39079a64.slice/crio-f326d3007ca6b59d57d59320b084c36a92298ae65a10c7bbea0588b94e6ecdaf WatchSource:0}: Error finding container f326d3007ca6b59d57d59320b084c36a92298ae65a10c7bbea0588b94e6ecdaf: Status 404 returned error can't find the container with id f326d3007ca6b59d57d59320b084c36a92298ae65a10c7bbea0588b94e6ecdaf
Nov 04 00:02:58.120126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.120096    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"]
Nov 04 00:02:58.165400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.165371    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"]
Nov 04 00:02:58.187557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.187530    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.198330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.197486    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"]
Nov 04 00:02:58.207928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.204540    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-06ade4-dockercfg-d842v\""
Nov 04 00:02:58.207928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.204589    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-06ade4\""
Nov 04 00:02:58.288811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.286706    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.288811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.286838    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hc6\" (UniqueName: \"kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6\") pod \"payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.397048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.392928    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"]
Nov 04 00:02:58.556155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.556086    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.556303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.556184    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hc6\" (UniqueName: \"kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6\") pod \"payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.556869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.556849    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.570284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.570260    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hc6\" (UniqueName: \"kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6\") pod \"payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:58.810485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:58.810395    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:02:59.056489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.056362    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" event={"ID":"ad917029-10ba-40c9-9b87-988c39079a64","Type":"ContainerStarted","Data":"f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e"}
Nov 04 00:02:59.056489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.056424    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" event={"ID":"ad917029-10ba-40c9-9b87-988c39079a64","Type":"ContainerStarted","Data":"f326d3007ca6b59d57d59320b084c36a92298ae65a10c7bbea0588b94e6ecdaf"}
Nov 04 00:02:59.056489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.056491    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:02:59.058111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.058083    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" event={"ID":"36656517-378e-4358-b97f-1fd76762cf94","Type":"ContainerStarted","Data":"ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0"}
Nov 04 00:02:59.058251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.058139    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" event={"ID":"36656517-378e-4358-b97f-1fd76762cf94","Type":"ContainerStarted","Data":"8ecae5f9f9b18ed2f0958a5670a0de21063d10e6b3b9b75899f02d2ef5a9cdbe"}
Nov 04 00:02:59.058251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.058170    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:02:59.087205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.087075    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" podStartSLOduration=2.087056456 podStartE2EDuration="2.087056456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:59.085799039 +0000 UTC m=+2044.185399183" watchObservedRunningTime="2026-11-04 00:02:59.087056456 +0000 UTC m=+2044.186656605"
Nov 04 00:02:59.107766 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:02:59.107714    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07237989_93da_496c_8fe0_60e681a60ded.slice/crio-bbc75ea9108afcec5fd1c99e3739ef0b00a1acadf003260c4ea4231c438dc5ff WatchSource:0}: Error finding container bbc75ea9108afcec5fd1c99e3739ef0b00a1acadf003260c4ea4231c438dc5ff: Status 404 returned error can't find the container with id bbc75ea9108afcec5fd1c99e3739ef0b00a1acadf003260c4ea4231c438dc5ff
Nov 04 00:02:59.107958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.107934    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"]
Nov 04 00:02:59.126183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:02:59.126124    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" podStartSLOduration=2.126109574 podStartE2EDuration="2.126109574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:59.112526856 +0000 UTC m=+2044.212127001" watchObservedRunningTime="2026-11-04 00:02:59.126109574 +0000 UTC m=+2044.225709718"
Nov 04 00:03:00.068568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:00.068517    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" event={"ID":"07237989-93da-496c-8fe0-60e681a60ded","Type":"ContainerStarted","Data":"463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd"}
Nov 04 00:03:00.069108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:00.068590    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" event={"ID":"07237989-93da-496c-8fe0-60e681a60ded","Type":"ContainerStarted","Data":"bbc75ea9108afcec5fd1c99e3739ef0b00a1acadf003260c4ea4231c438dc5ff"}
Nov 04 00:03:00.069108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:00.068989    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:03:00.102727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:00.102667    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" podStartSLOduration=2.102653791 podStartE2EDuration="2.102653791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:00.098148259 +0000 UTC m=+2045.197748403" watchObservedRunningTime="2026-11-04 00:03:00.102653791 +0000 UTC m=+2045.202253934"
Nov 04 00:03:10.072131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:10.072096    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:03:10.072617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:10.072174    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:03:11.079853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:11.079821    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:03:28.024354 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:03:28.024315    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:03:44.224099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.224065    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"]
Nov 04 00:03:44.231591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.231561    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.239640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.239611    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"]
Nov 04 00:03:44.243008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.242988    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"openshift-service-ca.crt\""
Nov 04 00:03:44.244728 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.244709    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2697a0\""
Nov 04 00:03:44.250720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.250695    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"kube-root-ca.crt\""
Nov 04 00:03:44.250720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.250703    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"routing-overlay\""
Nov 04 00:03:44.271793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.271370    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.271793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.271559    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcp6b\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.271793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.271699    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.271986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.271835    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.308318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.308292    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-58159406142a\""
Nov 04 00:03:44.310872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.310855    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-acf8cd35b468\""
Nov 04 00:03:44.310961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.310860    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-9cbba8262063\""
Nov 04 00:03:44.310961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.310911    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-e2e--5d63ff17-dockercfg-lrbbg\""
Nov 04 00:03:44.373407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.373371    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.373556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.373449    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.373615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.373603    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.373731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.373712    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcp6b\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.374246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.374224    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.374375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.374354    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.376235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.376214    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.387450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.387427    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcp6b\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.550393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.550308    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:44.845200 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:03:44.845163    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9e62d6_f387_4544_89f7_94a8af8cfb15.slice/crio-ba122cb25c4e6cffa624a022fa01cd7fc90ac21b4862ec2508f87080771e1d5c WatchSource:0}: Error finding container ba122cb25c4e6cffa624a022fa01cd7fc90ac21b4862ec2508f87080771e1d5c: Status 404 returned error can't find the container with id ba122cb25c4e6cffa624a022fa01cd7fc90ac21b4862ec2508f87080771e1d5c
Nov 04 00:03:44.847655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:44.847630    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"]
Nov 04 00:03:45.620763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:45.620713    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs" event={"ID":"aa9e62d6-f387-4544-89f7-94a8af8cfb15","Type":"ContainerStarted","Data":"d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d"}
Nov 04 00:03:45.621263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:45.621207    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:45.621263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:45.621249    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs" event={"ID":"aa9e62d6-f387-4544-89f7-94a8af8cfb15","Type":"ContainerStarted","Data":"ba122cb25c4e6cffa624a022fa01cd7fc90ac21b4862ec2508f87080771e1d5c"}
Nov 04 00:03:45.649804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:45.649754    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs" podStartSLOduration=1.6497274100000001 podStartE2EDuration="1.64972741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:45.647927419 +0000 UTC m=+2090.747527580" watchObservedRunningTime="2026-11-04 00:03:45.64972741 +0000 UTC m=+2090.749327554"
Nov 04 00:03:55.560656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:55.560618    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:03:55.618931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:55.618891    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:03:56.632830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:03:56.632785    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:03:58.025694 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:03:58.025659    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:04:05.033354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:05.033320    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"]
Nov 04 00:04:05.041083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:05.041058    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2c0c81-727hf"]
Nov 04 00:04:07.023637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:07.023597    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd71f8d-6d18-4d1b-a0e6-7c6751057cac" path="/var/lib/kubelet/pods/fcd71f8d-6d18-4d1b-a0e6-7c6751057cac/volumes"
Nov 04 00:04:16.046033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:16.045990    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"]
Nov 04 00:04:16.050063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:16.050036    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01cce37-grlhg"]
Nov 04 00:04:17.023563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:17.023519    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc85bf65-dbd9-428e-854d-4984fa6c0926" path="/var/lib/kubelet/pods/dc85bf65-dbd9-428e-854d-4984fa6c0926/volumes"
Nov 04 00:04:21.858348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.858312    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"]
Nov 04 00:04:21.863061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.863040    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:21.872526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.872499    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"agc-cred-6e0624cf5ba7\""
Nov 04 00:04:21.872872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.872791    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"]
Nov 04 00:04:21.873082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.872820    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"kube-root-ca.crt\""
Nov 04 00:04:21.873082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.872895    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"routing-overlay\""
Nov 04 00:04:21.877891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.877864    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-06ade4\""
Nov 04 00:04:21.878148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.878134    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"openshift-service-ca.crt\""
Nov 04 00:04:21.880419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.880396    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"payload-processing-external-model-e2e--f0c21228-dockercfg-5pjtp\""
Nov 04 00:04:21.986408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.983262    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:21.986408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.983393    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:21.986408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.983474    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:21.986408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:21.983527    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czqz\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.161042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.160953    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.161946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.161878    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czqz\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.162499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.162253    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.162499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.162409    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.163773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.163306    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.165035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.164501    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.185756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.185708    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.189197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.189162    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czqz\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.189592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.189553    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:22.517446 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:04:22.517410    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe99feb_286e_41c3_98a4_d8f35d72eac9.slice/crio-87b0d9d6ef4850ec263b074bf63c6a0e32e9f946ccb4471870c19380f32ab00f WatchSource:0}: Error finding container 87b0d9d6ef4850ec263b074bf63c6a0e32e9f946ccb4471870c19380f32ab00f: Status 404 returned error can't find the container with id 87b0d9d6ef4850ec263b074bf63c6a0e32e9f946ccb4471870c19380f32ab00f
Nov 04 00:04:22.518618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:22.518588    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"]
Nov 04 00:04:23.147839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:23.147796    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn" event={"ID":"bbe99feb-286e-41c3-98a4-d8f35d72eac9","Type":"ContainerStarted","Data":"82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54"}
Nov 04 00:04:23.148271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:23.147860    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn" event={"ID":"bbe99feb-286e-41c3-98a4-d8f35d72eac9","Type":"ContainerStarted","Data":"87b0d9d6ef4850ec263b074bf63c6a0e32e9f946ccb4471870c19380f32ab00f"}
Nov 04 00:04:23.148271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:23.147910    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:23.179662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:23.179608    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn" podStartSLOduration=2.179596435 podStartE2EDuration="2.179596435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:04:23.176306725 +0000 UTC m=+2128.275906868" watchObservedRunningTime="2026-11-04 00:04:23.179596435 +0000 UTC m=+2128.279196579"
Nov 04 00:04:25.104094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.104054    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"]
Nov 04 00:04:25.109333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.109306    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.115355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.115323    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"]
Nov 04 00:04:25.131013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.130990    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b4376d\""
Nov 04 00:04:25.145223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.145201    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b4376d-dockercfg-5kq5s\""
Nov 04 00:04:25.242841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.224263    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.242841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.224437    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6xx\" (UniqueName: \"kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx\") pod \"payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.333259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.333215    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.333472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.333384    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6xx\" (UniqueName: \"kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx\") pod \"payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.347803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.342113    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.354478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.354403    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6xx\" (UniqueName: \"kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx\") pod \"payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.430842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.430812    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:25.741855 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:04:25.741817    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac22190_7677_4194_9c52_47b04738f995.slice/crio-5b2241f755799ea192a7a731680a3c98125fa30dc12f4a2c311fa2ccbd3900ac WatchSource:0}: Error finding container 5b2241f755799ea192a7a731680a3c98125fa30dc12f4a2c311fa2ccbd3900ac: Status 404 returned error can't find the container with id 5b2241f755799ea192a7a731680a3c98125fa30dc12f4a2c311fa2ccbd3900ac
Nov 04 00:04:25.742459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:25.742437    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"]
Nov 04 00:04:26.187995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:26.187952    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" event={"ID":"1ac22190-7677-4194-9c52-47b04738f995","Type":"ContainerStarted","Data":"e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce"}
Nov 04 00:04:26.188547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:26.188023    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" event={"ID":"1ac22190-7677-4194-9c52-47b04738f995","Type":"ContainerStarted","Data":"5b2241f755799ea192a7a731680a3c98125fa30dc12f4a2c311fa2ccbd3900ac"}
Nov 04 00:04:26.188547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:26.188171    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:26.218545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:26.218485    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" podStartSLOduration=1.218470942 podStartE2EDuration="1.218470942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:04:26.214203562 +0000 UTC m=+2131.313803706" watchObservedRunningTime="2026-11-04 00:04:26.218470942 +0000 UTC m=+2131.318071085"
Nov 04 00:04:28.024536 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:04:28.024489    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:04:31.145803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.145761    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"]
Nov 04 00:04:31.154761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.151194    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.167779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.166211    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"]
Nov 04 00:04:31.167779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.167291    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\""
Nov 04 00:04:31.167779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.167316    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\""
Nov 04 00:04:31.174041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.174002    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\""
Nov 04 00:04:31.176813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.176776    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\""
Nov 04 00:04:31.176986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.176966    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-261bd6139428\""
Nov 04 00:04:31.179811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.179786    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-hsvcw\""
Nov 04 00:04:31.182529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.182508    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-d87e354e0f00\""
Nov 04 00:04:31.224723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.224668    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.224932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.224811    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.225040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.225010    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxwj\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.225119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.225105    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.327015 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.326976    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxwj\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.327218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.327023    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.327218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.327113    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.327385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.327242    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.328097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.328056    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.328239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.328136    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.330332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.330308    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.358603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.358580    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxwj\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.479255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.479225    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:31.822520 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:04:31.822488    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf152b4_2808_4179_b550_51fff15757e3.slice/crio-0a23051600c3fee78f8840b8839b7a6d7d5784fafe0eaff3fbe3b485b09b8cc7 WatchSource:0}: Error finding container 0a23051600c3fee78f8840b8839b7a6d7d5784fafe0eaff3fbe3b485b09b8cc7: Status 404 returned error can't find the container with id 0a23051600c3fee78f8840b8839b7a6d7d5784fafe0eaff3fbe3b485b09b8cc7
Nov 04 00:04:31.827530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:31.827503    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"]
Nov 04 00:04:32.252179 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:32.252136    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv" event={"ID":"faf152b4-2808-4179-b550-51fff15757e3","Type":"ContainerStarted","Data":"c2fc657e506f7b8a17f40a18b88792d2fb374cc5798957fd01bc4fc734cc93dc"}
Nov 04 00:04:32.252658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:32.252235    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:32.252658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:32.252257    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv" event={"ID":"faf152b4-2808-4179-b550-51fff15757e3","Type":"ContainerStarted","Data":"0a23051600c3fee78f8840b8839b7a6d7d5784fafe0eaff3fbe3b485b09b8cc7"}
Nov 04 00:04:32.287560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:32.287506    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv" podStartSLOduration=1.2874928589999999 podStartE2EDuration="1.287492859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:04:32.279093572 +0000 UTC m=+2137.378693716" watchObservedRunningTime="2026-11-04 00:04:32.287492859 +0000 UTC m=+2137.387093041"
Nov 04 00:04:34.160530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:34.160499    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:04:37.199725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:37.199690    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:04:43.264446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:43.264416    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:04:55.033470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:55.033433    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"]
Nov 04 00:04:55.039084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:55.039056    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26090e3c-dcv89"]
Nov 04 00:04:57.022895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:57.022859    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a394d1c2-c8da-4b5d-af3f-d89edb03dd0f" path="/var/lib/kubelet/pods/a394d1c2-c8da-4b5d-af3f-d89edb03dd0f/volumes"
Nov 04 00:04:57.075514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:57.075468    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"]
Nov 04 00:04:57.080250 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:57.080223    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72c10c-slw6x"]
Nov 04 00:04:58.027978 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:04:58.027934    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:04:59.023468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:04:59.023425    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6aa3249-0fc9-4c24-93fd-c51e3dccc018" path="/var/lib/kubelet/pods/c6aa3249-0fc9-4c24-93fd-c51e3dccc018/volumes"
Nov 04 00:05:08.040735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:08.040690    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"]
Nov 04 00:05:08.063004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:08.062967    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194475f0-7k7rm"]
Nov 04 00:05:09.076589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:09.076543    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2e92fa-4fbe-44b8-a3f6-b700da748344" path="/var/lib/kubelet/pods/5f2e92fa-4fbe-44b8-a3f6-b700da748344/volumes"
Nov 04 00:05:17.327469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:17.327429    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"]
Nov 04 00:05:22.328049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.328004    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs" podUID="aa9e62d6-f387-4544-89f7-94a8af8cfb15" containerName="payload-processing" containerID="cri-o://d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d" gracePeriod=25
Nov 04 00:05:22.620278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.620247    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:05:22.652642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.652608    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials\") pod \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") "
Nov 04 00:05:22.652864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.652704    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b\") pod \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") "
Nov 04 00:05:22.652864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.652828    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume\") pod \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") "
Nov 04 00:05:22.653005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.652866    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume\") pod \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\" (UID: \"aa9e62d6-f387-4544-89f7-94a8af8cfb15\") "
Nov 04 00:05:22.653942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.653908    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume" pod "aa9e62d6-f387-4544-89f7-94a8af8cfb15" (UID: "aa9e62d6-f387-4544-89f7-94a8af8cfb15"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:22.654330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.654306    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume" pod "aa9e62d6-f387-4544-89f7-94a8af8cfb15" (UID: "aa9e62d6-f387-4544-89f7-94a8af8cfb15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:22.656801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.656774    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials" pod "aa9e62d6-f387-4544-89f7-94a8af8cfb15" (UID: "aa9e62d6-f387-4544-89f7-94a8af8cfb15"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:22.660685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.660657    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b" pod "aa9e62d6-f387-4544-89f7-94a8af8cfb15" (UID: "aa9e62d6-f387-4544-89f7-94a8af8cfb15"). InnerVolumeSpecName "kube-api-access-mcp6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:22.761359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.761286    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:22.761568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.761388    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcp6b\" (UniqueName: \"kubernetes.io/projected/aa9e62d6-f387-4544-89f7-94a8af8cfb15-kube-api-access-mcp6b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:22.761568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.761408    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:22.761568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.761423    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa9e62d6-f387-4544-89f7-94a8af8cfb15-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:22.799836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.799795    5074 generic.go:356] "Generic (PLEG): container finished" podID="aa9e62d6-f387-4544-89f7-94a8af8cfb15" containerID="d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d" exitCode=0
Nov 04 00:05:22.800028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.799852    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"
Nov 04 00:05:22.800028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.799892    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs" event={"ID":"aa9e62d6-f387-4544-89f7-94a8af8cfb15","Type":"ContainerDied","Data":"d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d"}
Nov 04 00:05:22.800028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.799991    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs" event={"ID":"aa9e62d6-f387-4544-89f7-94a8af8cfb15","Type":"ContainerDied","Data":"ba122cb25c4e6cffa624a022fa01cd7fc90ac21b4862ec2508f87080771e1d5c"}
Nov 04 00:05:22.800028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.800022    5074 scope.go:122] "RemoveContainer" containerID="d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d"
Nov 04 00:05:22.821250 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.821221    5074 scope.go:122] "RemoveContainer" containerID="d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d"
Nov 04 00:05:22.821558 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:05:22.821535    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d\": container with ID starting with d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d not found: ID does not exist" containerID="d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d"
Nov 04 00:05:22.821637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.821571    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d"} err="failed to get container status \"d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d\": rpc error: code = NotFound desc = could not find container \"d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d\": container with ID starting with d38994699b526eb9ab11dcc884843ca3c36cc91670bc8226779e2d892bae214d not found: ID does not exist"
Nov 04 00:05:22.839945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.839881    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"]
Nov 04 00:05:22.848891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:22.848861    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-98b9xxphs"]
Nov 04 00:05:23.057402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:23.057365    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9e62d6-f387-4544-89f7-94a8af8cfb15" path="/var/lib/kubelet/pods/aa9e62d6-f387-4544-89f7-94a8af8cfb15/volumes"
Nov 04 00:05:24.031764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.023565    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"]
Nov 04 00:05:24.031764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.024882    5074 cpu_manager.go:414] "containerMap: removing container" podUID="aa9e62d6-f387-4544-89f7-94a8af8cfb15" containerName="payload-processing"
Nov 04 00:05:24.031764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.024902    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9e62d6-f387-4544-89f7-94a8af8cfb15" containerName="payload-processing"
Nov 04 00:05:24.031764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.025131    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9e62d6-f387-4544-89f7-94a8af8cfb15" containerName="payload-processing"
Nov 04 00:05:24.039760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.034648    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.059774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.055670    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-acf8cd35b468\""
Nov 04 00:05:24.069803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.068995    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"]
Nov 04 00:05:24.078804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.078702    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-e2e--5d63ff17-dockercfg-lrbbg\""
Nov 04 00:05:24.078992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.078851    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"kube-root-ca.crt\""
Nov 04 00:05:24.079059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.078994    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"routing-overlay\""
Nov 04 00:05:24.079059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.079054    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"openshift-service-ca.crt\""
Nov 04 00:05:24.079178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.079146    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2697a0\""
Nov 04 00:05:24.093969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.093918    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.094456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.094412    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.095884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.094623    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.095884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.094799    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkkt\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.196695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.196656    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.196695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.196696    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.197086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.196811    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkkt\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.197086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.196980    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.197505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.197482    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.197636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.197615    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.199677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.199649    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.213159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.213125    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkkt\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.409583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.409498    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:24.964926 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:05:24.964890    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d718c8_a83d_4b45_a1f5_c4fae9965f79.slice/crio-895e40aa6e5f03de6ba3a795a3e27a40c3815f1f20ea57fd8827edf087c26365 WatchSource:0}: Error finding container 895e40aa6e5f03de6ba3a795a3e27a40c3815f1f20ea57fd8827edf087c26365: Status 404 returned error can't find the container with id 895e40aa6e5f03de6ba3a795a3e27a40c3815f1f20ea57fd8827edf087c26365
Nov 04 00:05:24.967511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:24.967478    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"]
Nov 04 00:05:25.729415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:25.728993    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"]
Nov 04 00:05:25.886439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:25.886391    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq" event={"ID":"25d718c8-a83d-4b45-a1f5-c4fae9965f79","Type":"ContainerStarted","Data":"5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2"}
Nov 04 00:05:25.886618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:25.886465    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq" event={"ID":"25d718c8-a83d-4b45-a1f5-c4fae9965f79","Type":"ContainerStarted","Data":"895e40aa6e5f03de6ba3a795a3e27a40c3815f1f20ea57fd8827edf087c26365"}
Nov 04 00:05:25.886618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:25.886493    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:26.037557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:26.037440    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq" podStartSLOduration=3.037420452 podStartE2EDuration="3.037420452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:25.938510412 +0000 UTC m=+2191.038110557" watchObservedRunningTime="2026-11-04 00:05:26.037420452 +0000 UTC m=+2191.137020595"
Nov 04 00:05:26.040319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:26.040293    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"]
Nov 04 00:05:26.047035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:26.046936    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8ff953-rd5rs"]
Nov 04 00:05:27.039539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:27.039500    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e13b562-d3c7-484f-a114-d33a6f5cd895" path="/var/lib/kubelet/pods/4e13b562-d3c7-484f-a114-d33a6f5cd895/volumes"
Nov 04 00:05:28.023812 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:05:28.023769    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:05:30.729826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.729780    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv" podUID="faf152b4-2808-4179-b550-51fff15757e3" containerName="payload-processing" containerID="cri-o://c2fc657e506f7b8a17f40a18b88792d2fb374cc5798957fd01bc4fc734cc93dc" gracePeriod=25
Nov 04 00:05:30.877930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.875044    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"]
Nov 04 00:05:30.918547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.912636    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"]
Nov 04 00:05:30.918547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.912687    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"]
Nov 04 00:05:30.918547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.912760    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:30.918920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.918799    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"]
Nov 04 00:05:30.918920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.918905    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:30.921332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.921303    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c7d218\""
Nov 04 00:05:30.934025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.931877    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c7d218-dockercfg-fpjpx\""
Nov 04 00:05:30.940025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.939999    5074 generic.go:356] "Generic (PLEG): container finished" podID="faf152b4-2808-4179-b550-51fff15757e3" containerID="c2fc657e506f7b8a17f40a18b88792d2fb374cc5798957fd01bc4fc734cc93dc" exitCode=0
Nov 04 00:05:30.940265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:30.940246    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv" event={"ID":"faf152b4-2808-4179-b550-51fff15757e3","Type":"ContainerDied","Data":"c2fc657e506f7b8a17f40a18b88792d2fb374cc5798957fd01bc4fc734cc93dc"}
Nov 04 00:05:31.148785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.147139    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.148785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.147734    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229pg\" (UniqueName: \"kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg\") pod \"payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.148785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.147997    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcpr\" (UniqueName: \"kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr\") pod \"payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.148785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.148052    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.191491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.189203    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:05:31.270733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.270686    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj\") pod \"faf152b4-2808-4179-b550-51fff15757e3\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") "
Nov 04 00:05:31.271086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271063    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials\") pod \"faf152b4-2808-4179-b550-51fff15757e3\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") "
Nov 04 00:05:31.271200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271157    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume\") pod \"faf152b4-2808-4179-b550-51fff15757e3\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") "
Nov 04 00:05:31.271269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271207    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume\") pod \"faf152b4-2808-4179-b550-51fff15757e3\" (UID: \"faf152b4-2808-4179-b550-51fff15757e3\") "
Nov 04 00:05:31.271809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271481    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcpr\" (UniqueName: \"kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr\") pod \"payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.271809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271535    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.271809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271779    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.272043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.271879    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229pg\" (UniqueName: \"kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg\") pod \"payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.282228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.280347    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume" pod "faf152b4-2808-4179-b550-51fff15757e3" (UID: "faf152b4-2808-4179-b550-51fff15757e3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:31.282228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.281251    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.282228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.281358    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.282228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.281860    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume" pod "faf152b4-2808-4179-b550-51fff15757e3" (UID: "faf152b4-2808-4179-b550-51fff15757e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:31.302836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.298548    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcpr\" (UniqueName: \"kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr\") pod \"payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.318655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.318375    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj" pod "faf152b4-2808-4179-b550-51fff15757e3" (UID: "faf152b4-2808-4179-b550-51fff15757e3"). InnerVolumeSpecName "kube-api-access-mcxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:31.318655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.318598    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229pg\" (UniqueName: \"kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg\") pod \"payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.320689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.320653    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials" pod "faf152b4-2808-4179-b550-51fff15757e3" (UID: "faf152b4-2808-4179-b550-51fff15757e3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:31.373924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.373824    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:31.373924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.373856    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:31.373924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.373866    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/faf152b4-2808-4179-b550-51fff15757e3-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:31.373924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.373876    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcxwj\" (UniqueName: \"kubernetes.io/projected/faf152b4-2808-4179-b550-51fff15757e3-kube-api-access-mcxwj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:31.553214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.553179    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:31.563939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.563914    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:31.854107 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:05:31.854064    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb160dbb1_fdd5_4d63_add3_03fb9bdebbcc.slice/crio-7760df4e305ad3b67a3215b0874ed38563ffe52b5f3127f34669f7660f3d6b8a WatchSource:0}: Error finding container 7760df4e305ad3b67a3215b0874ed38563ffe52b5f3127f34669f7660f3d6b8a: Status 404 returned error can't find the container with id 7760df4e305ad3b67a3215b0874ed38563ffe52b5f3127f34669f7660f3d6b8a
Nov 04 00:05:31.857952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.857930    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"]
Nov 04 00:05:31.881958 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:05:31.881924    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dbd136_817b_4a31_85c8_be66c017b81d.slice/crio-54c50e7ad7ecc36f010cf55ddb2d7be9d77e55ec0cc706973d36c69c2fc8ab30 WatchSource:0}: Error finding container 54c50e7ad7ecc36f010cf55ddb2d7be9d77e55ec0cc706973d36c69c2fc8ab30: Status 404 returned error can't find the container with id 54c50e7ad7ecc36f010cf55ddb2d7be9d77e55ec0cc706973d36c69c2fc8ab30
Nov 04 00:05:31.882496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.882473    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"]
Nov 04 00:05:31.957794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.957719    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht" event={"ID":"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc","Type":"ContainerStarted","Data":"7760df4e305ad3b67a3215b0874ed38563ffe52b5f3127f34669f7660f3d6b8a"}
Nov 04 00:05:31.960011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.959983    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" event={"ID":"85dbd136-817b-4a31-85c8-be66c017b81d","Type":"ContainerStarted","Data":"54c50e7ad7ecc36f010cf55ddb2d7be9d77e55ec0cc706973d36c69c2fc8ab30"}
Nov 04 00:05:31.961642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.961613    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv" event={"ID":"faf152b4-2808-4179-b550-51fff15757e3","Type":"ContainerDied","Data":"0a23051600c3fee78f8840b8839b7a6d7d5784fafe0eaff3fbe3b485b09b8cc7"}
Nov 04 00:05:31.961805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.961665    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"
Nov 04 00:05:31.961805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:31.961671    5074 scope.go:122] "RemoveContainer" containerID="c2fc657e506f7b8a17f40a18b88792d2fb374cc5798957fd01bc4fc734cc93dc"
Nov 04 00:05:32.007960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.002701    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"]
Nov 04 00:05:32.024056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.024005    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9cr6wv"]
Nov 04 00:05:32.035762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.035337    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"]
Nov 04 00:05:32.042343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.042307    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c84efd-hmt8h"]
Nov 04 00:05:32.971385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.971342    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht" event={"ID":"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc","Type":"ContainerStarted","Data":"7c3a41c98d2a04232112d7371289f2a68cb75d008c82bbbb6f28e44d97f4f77a"}
Nov 04 00:05:32.971834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.971451    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:32.973237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.973207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" event={"ID":"85dbd136-817b-4a31-85c8-be66c017b81d","Type":"ContainerStarted","Data":"f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903"}
Nov 04 00:05:32.973381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:32.973275    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:33.006979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.006925    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht" podStartSLOduration=3.006909751 podStartE2EDuration="3.006909751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:33.000587358 +0000 UTC m=+2198.100187533" watchObservedRunningTime="2026-11-04 00:05:33.006909751 +0000 UTC m=+2198.106509896"
Nov 04 00:05:33.021619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.021585    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ae30cd-734b-4c8d-82c8-716b83cbca4e" path="/var/lib/kubelet/pods/38ae30cd-734b-4c8d-82c8-716b83cbca4e/volumes"
Nov 04 00:05:33.022319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.022298    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf152b4-2808-4179-b550-51fff15757e3" path="/var/lib/kubelet/pods/faf152b4-2808-4179-b550-51fff15757e3/volumes"
Nov 04 00:05:33.256342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.256243    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" podStartSLOduration=3.256229533 podStartE2EDuration="3.256229533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:33.253112312 +0000 UTC m=+2198.352712455" watchObservedRunningTime="2026-11-04 00:05:33.256229533 +0000 UTC m=+2198.355829676"
Nov 04 00:05:33.845459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.845420    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"]
Nov 04 00:05:33.846312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.846291    5074 cpu_manager.go:414] "containerMap: removing container" podUID="faf152b4-2808-4179-b550-51fff15757e3" containerName="payload-processing"
Nov 04 00:05:33.846312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.846309    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf152b4-2808-4179-b550-51fff15757e3" containerName="payload-processing"
Nov 04 00:05:33.846508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.846435    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf152b4-2808-4179-b550-51fff15757e3" containerName="payload-processing"
Nov 04 00:05:33.851524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.851504    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:33.860434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.859946    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"]
Nov 04 00:05:33.873049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.868696    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:05:33.882568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.876943    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:05:33.915471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.915437    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl4s\" (UniqueName: \"kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:33.915623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.915594    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:33.915814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:33.915782    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.017677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.017638    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl4s\" (UniqueName: \"kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.018167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.017735    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.018167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.017869    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.018526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.018494    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.020256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.020235    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.031152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.031128    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl4s\" (UniqueName: \"kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s\") pod \"maas-api-revoke-sub-23153751-gnw4m\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.171505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.171422    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:34.665969 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:05:34.665938    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c7f9f1_299f_4b53_81d7_37ee15c7c5d3.slice/crio-014bfc413cba5c5a13a5f3e4bf2a4b8781086125e11d5752463d3dfc590825c4 WatchSource:0}: Error finding container 014bfc413cba5c5a13a5f3e4bf2a4b8781086125e11d5752463d3dfc590825c4: Status 404 returned error can't find the container with id 014bfc413cba5c5a13a5f3e4bf2a4b8781086125e11d5752463d3dfc590825c4
Nov 04 00:05:34.681193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.681167    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"]
Nov 04 00:05:34.992324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:34.992288    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m" event={"ID":"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3","Type":"ContainerStarted","Data":"014bfc413cba5c5a13a5f3e4bf2a4b8781086125e11d5752463d3dfc590825c4"}
Nov 04 00:05:36.008291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:36.008249    5074 generic.go:356] "Generic (PLEG): container finished" podID="75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" containerID="ae3da5d4c5d0e80f0d4d48c8a320472df0b9c0fffde31c373e356d43e3707291" exitCode=0
Nov 04 00:05:36.008797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:36.008338    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m" event={"ID":"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3","Type":"ContainerDied","Data":"ae3da5d4c5d0e80f0d4d48c8a320472df0b9c0fffde31c373e356d43e3707291"}
Nov 04 00:05:36.899258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:36.899228    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:05:37.195519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.195493    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:37.309258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.309226    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s\" (UniqueName: \"kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s\") pod \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") "
Nov 04 00:05:37.309431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.309394    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca\") pod \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") "
Nov 04 00:05:37.309485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.309461    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca\") pod \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\" (UID: \"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3\") "
Nov 04 00:05:37.309838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.309809    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca" pod "75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" (UID: "75c7f9f1-299f-4b53-81d7-37ee15c7c5d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:37.310115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.310093    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:37.311656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.311628    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca" pod "75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" (UID: "75c7f9f1-299f-4b53-81d7-37ee15c7c5d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:05:37.312003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.311983    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s" pod "75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" (UID: "75c7f9f1-299f-4b53-81d7-37ee15c7c5d3"). InnerVolumeSpecName "kube-api-access-czl4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:37.412593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.412513    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czl4s\" (UniqueName: \"kubernetes.io/projected/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-kube-api-access-czl4s\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:37.412593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:37.412553    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:38.027465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:38.027431    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m" event={"ID":"75c7f9f1-299f-4b53-81d7-37ee15c7c5d3","Type":"ContainerDied","Data":"014bfc413cba5c5a13a5f3e4bf2a4b8781086125e11d5752463d3dfc590825c4"}
Nov 04 00:05:38.027772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:38.027728    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="014bfc413cba5c5a13a5f3e4bf2a4b8781086125e11d5752463d3dfc590825c4"
Nov 04 00:05:38.027886 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:38.027463    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"
Nov 04 00:05:42.306376 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.306331    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"]
Nov 04 00:05:42.307698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.307647    5074 cpu_manager.go:414] "containerMap: removing container" podUID="75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" containerName="revoke-keys"
Nov 04 00:05:42.307698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.307676    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" containerName="revoke-keys"
Nov 04 00:05:42.307972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.307955    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" containerName="revoke-keys"
Nov 04 00:05:42.312848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.312784    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"]
Nov 04 00:05:42.313003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.312876    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.327082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.327059    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:05:42.328529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.328494    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:05:42.412161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.412116    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.413147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.413116    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z29gg\" (UniqueName: \"kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.413355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.413325    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.514812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.514770    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.515007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.514981    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.515100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.515074    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z29gg\" (UniqueName: \"kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.515954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.515924    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.517405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.517376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.536111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.536085    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z29gg\" (UniqueName: \"kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg\") pod \"maas-api-revoke-sub-48229cab-lb7jq\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.631192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.631107    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:42.917630 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:05:42.917595    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6636a4d9_8c65_49d3_a995_caf08906a24e.slice/crio-79cdf1bad74696a826c91400e9c71b8cf0219e2de60655163f61765e579b217e WatchSource:0}: Error finding container 79cdf1bad74696a826c91400e9c71b8cf0219e2de60655163f61765e579b217e: Status 404 returned error can't find the container with id 79cdf1bad74696a826c91400e9c71b8cf0219e2de60655163f61765e579b217e
Nov 04 00:05:42.921610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:42.921584    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"]
Nov 04 00:05:43.047944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:43.047893    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"]
Nov 04 00:05:43.052042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:43.052016    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e2c9ce-lhcrs"]
Nov 04 00:05:43.071751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:43.071708    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq" event={"ID":"6636a4d9-8c65-49d3-a995-caf08906a24e","Type":"ContainerStarted","Data":"79cdf1bad74696a826c91400e9c71b8cf0219e2de60655163f61765e579b217e"}
Nov 04 00:05:43.985774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:43.985721    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:05:43.986240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:43.985806    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:05:44.084286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.084247    5074 generic.go:356] "Generic (PLEG): container finished" podID="6636a4d9-8c65-49d3-a995-caf08906a24e" containerID="1b4ff6fb2b2714b5f006450f0597f0873f1f397e9e8feff270d1c4318c27852b" exitCode=0
Nov 04 00:05:44.084456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.084296    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq" event={"ID":"6636a4d9-8c65-49d3-a995-caf08906a24e","Type":"ContainerDied","Data":"1b4ff6fb2b2714b5f006450f0597f0873f1f397e9e8feff270d1c4318c27852b"}
Nov 04 00:05:44.869125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.869090    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"]
Nov 04 00:05:44.874049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.873997    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:44.882000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.881972    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"]
Nov 04 00:05:44.963764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.963106    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjgt\" (UniqueName: \"kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:44.963764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.963350    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:44.963764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:44.963425    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.029160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.029124    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed9152e-ee12-4c23-97d8-c2548cb6d695" path="/var/lib/kubelet/pods/9ed9152e-ee12-4c23-97d8-c2548cb6d695/volumes"
Nov 04 00:05:45.064555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.064516    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.064795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.064774    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjgt\" (UniqueName: \"kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.065165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.065140    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.065874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.065850    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.067516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.067492    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.078329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.078299    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjgt\" (UniqueName: \"kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt\") pod \"maas-api-revoke-sub-12179362-b6x7s\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.193132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.193100    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:45.285648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.285375    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:45.371667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.371625    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca\") pod \"6636a4d9-8c65-49d3-a995-caf08906a24e\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") "
Nov 04 00:05:45.371878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.371710    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg\" (UniqueName: \"kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg\") pod \"6636a4d9-8c65-49d3-a995-caf08906a24e\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") "
Nov 04 00:05:45.371878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.371845    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca\") pod \"6636a4d9-8c65-49d3-a995-caf08906a24e\" (UID: \"6636a4d9-8c65-49d3-a995-caf08906a24e\") "
Nov 04 00:05:45.372242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.372207    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca" pod "6636a4d9-8c65-49d3-a995-caf08906a24e" (UID: "6636a4d9-8c65-49d3-a995-caf08906a24e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:45.372611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.372590    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:45.373895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.373872    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg" pod "6636a4d9-8c65-49d3-a995-caf08906a24e" (UID: "6636a4d9-8c65-49d3-a995-caf08906a24e"). InnerVolumeSpecName "kube-api-access-z29gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:45.374080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.374059    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca" pod "6636a4d9-8c65-49d3-a995-caf08906a24e" (UID: "6636a4d9-8c65-49d3-a995-caf08906a24e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:05:45.474181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.474140    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6636a4d9-8c65-49d3-a995-caf08906a24e-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:45.474181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.474175    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z29gg\" (UniqueName: \"kubernetes.io/projected/6636a4d9-8c65-49d3-a995-caf08906a24e-kube-api-access-z29gg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:45.510483 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:05:45.510453    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dcc78a_5607_40cd_8db0_d71206b4ad7d.slice/crio-72f1195217b5876f0440b9ce776ad358b8682b75e2dafb142b3f1dd5660d29cc WatchSource:0}: Error finding container 72f1195217b5876f0440b9ce776ad358b8682b75e2dafb142b3f1dd5660d29cc: Status 404 returned error can't find the container with id 72f1195217b5876f0440b9ce776ad358b8682b75e2dafb142b3f1dd5660d29cc
Nov 04 00:05:45.513454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:45.513426    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"]
Nov 04 00:05:46.107976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:46.107942    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s" event={"ID":"59dcc78a-5607-40cd-8db0-d71206b4ad7d","Type":"ContainerStarted","Data":"34d7cf61eb8ec83374a65bf655a4595ff73426288a9e081f580aea6ce7fe0ecb"}
Nov 04 00:05:46.108411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:46.108007    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s" event={"ID":"59dcc78a-5607-40cd-8db0-d71206b4ad7d","Type":"ContainerStarted","Data":"72f1195217b5876f0440b9ce776ad358b8682b75e2dafb142b3f1dd5660d29cc"}
Nov 04 00:05:46.110063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:46.110038    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"
Nov 04 00:05:46.110063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:46.110056    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq" event={"ID":"6636a4d9-8c65-49d3-a995-caf08906a24e","Type":"ContainerDied","Data":"79cdf1bad74696a826c91400e9c71b8cf0219e2de60655163f61765e579b217e"}
Nov 04 00:05:46.110250 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:46.110103    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79cdf1bad74696a826c91400e9c71b8cf0219e2de60655163f61765e579b217e"
Nov 04 00:05:46.144864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:46.144809    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s" podStartSLOduration=2.144794851 podStartE2EDuration="2.144794851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:46.140749257 +0000 UTC m=+2211.240349396" watchObservedRunningTime="2026-11-04 00:05:46.144794851 +0000 UTC m=+2211.244394996"
Nov 04 00:05:47.124092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:47.123973    5074 generic.go:356] "Generic (PLEG): container finished" podID="59dcc78a-5607-40cd-8db0-d71206b4ad7d" containerID="34d7cf61eb8ec83374a65bf655a4595ff73426288a9e081f580aea6ce7fe0ecb" exitCode=0
Nov 04 00:05:47.162715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:47.124126    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s" event={"ID":"59dcc78a-5607-40cd-8db0-d71206b4ad7d","Type":"ContainerDied","Data":"34d7cf61eb8ec83374a65bf655a4595ff73426288a9e081f580aea6ce7fe0ecb"}
Nov 04 00:05:48.345537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.345506    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:48.424441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.424349    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca\") pod \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") "
Nov 04 00:05:48.424654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.424632    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt\" (UniqueName: \"kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt\") pod \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") "
Nov 04 00:05:48.424782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.424766    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca\") pod \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\" (UID: \"59dcc78a-5607-40cd-8db0-d71206b4ad7d\") "
Nov 04 00:05:48.425133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.425106    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca" pod "59dcc78a-5607-40cd-8db0-d71206b4ad7d" (UID: "59dcc78a-5607-40cd-8db0-d71206b4ad7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:48.425450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.425430    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:48.427318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.427284    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca" pod "59dcc78a-5607-40cd-8db0-d71206b4ad7d" (UID: "59dcc78a-5607-40cd-8db0-d71206b4ad7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:05:48.427631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.427598    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt" pod "59dcc78a-5607-40cd-8db0-d71206b4ad7d" (UID: "59dcc78a-5607-40cd-8db0-d71206b4ad7d"). InnerVolumeSpecName "kube-api-access-scjgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:48.526793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.526758    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjgt\" (UniqueName: \"kubernetes.io/projected/59dcc78a-5607-40cd-8db0-d71206b4ad7d-kube-api-access-scjgt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:48.526793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:48.526791    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dcc78a-5607-40cd-8db0-d71206b4ad7d-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:49.161755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:49.161707    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"
Nov 04 00:05:49.161959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:49.161732    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s" event={"ID":"59dcc78a-5607-40cd-8db0-d71206b4ad7d","Type":"ContainerDied","Data":"72f1195217b5876f0440b9ce776ad358b8682b75e2dafb142b3f1dd5660d29cc"}
Nov 04 00:05:49.161959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:49.161803    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f1195217b5876f0440b9ce776ad358b8682b75e2dafb142b3f1dd5660d29cc"
Nov 04 00:05:53.644579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:53.644507    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"]
Nov 04 00:05:58.023204 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:05:58.023168    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:05:58.546385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:58.546348    5074 scope.go:122] "RemoveContainer" containerID="298f6670f44b14d9255aebd9736e4e099e378ea041592e8b702a5618f8423d9f"
Nov 04 00:05:58.560473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:58.560449    5074 scope.go:122] "RemoveContainer" containerID="a2314cd60b8bfe7383db2d108dd03c53e5ffceb13559bcc928190ca4cf666673"
Nov 04 00:05:58.574271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:58.574241    5074 scope.go:122] "RemoveContainer" containerID="916914e51b8d95c01ebf22b0de0ba28aa1b3946f61e4ae39d19294f4366b45f2"
Nov 04 00:05:58.585478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:58.585430    5074 scope.go:122] "RemoveContainer" containerID="c5bcd078866ec5807d15084cf8d680f5df745ce74f3114b047461adc7cb64376"
Nov 04 00:05:58.652453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:58.652294    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn" podUID="bbe99feb-286e-41c3-98a4-d8f35d72eac9" containerName="payload-processing" containerID="cri-o://82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54" gracePeriod=25
Nov 04 00:05:59.161265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.161233    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:05:59.250773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.250720    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume\") pod \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") "
Nov 04 00:05:59.251020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.251001    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz\") pod \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") "
Nov 04 00:05:59.251217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.251086    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume\") pod \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") "
Nov 04 00:05:59.251324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.251227    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials\") pod \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\" (UID: \"bbe99feb-286e-41c3-98a4-d8f35d72eac9\") "
Nov 04 00:05:59.252600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.252566    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume" pod "bbe99feb-286e-41c3-98a4-d8f35d72eac9" (UID: "bbe99feb-286e-41c3-98a4-d8f35d72eac9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:59.252600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.252580    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume" pod "bbe99feb-286e-41c3-98a4-d8f35d72eac9" (UID: "bbe99feb-286e-41c3-98a4-d8f35d72eac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:59.255253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.255206    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz" pod "bbe99feb-286e-41c3-98a4-d8f35d72eac9" (UID: "bbe99feb-286e-41c3-98a4-d8f35d72eac9"). InnerVolumeSpecName "kube-api-access-4czqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:59.300409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.296929    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials" pod "bbe99feb-286e-41c3-98a4-d8f35d72eac9" (UID: "bbe99feb-286e-41c3-98a4-d8f35d72eac9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:59.356772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.355049    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:59.356772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.355087    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:59.356772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.355104    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czqz\" (UniqueName: \"kubernetes.io/projected/bbe99feb-286e-41c3-98a4-d8f35d72eac9-kube-api-access-4czqz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:59.356772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.355120    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe99feb-286e-41c3-98a4-d8f35d72eac9-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:59.409812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.403945    5074 generic.go:356] "Generic (PLEG): container finished" podID="bbe99feb-286e-41c3-98a4-d8f35d72eac9" containerID="82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54" exitCode=0
Nov 04 00:05:59.409812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.404118    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn" event={"ID":"bbe99feb-286e-41c3-98a4-d8f35d72eac9","Type":"ContainerDied","Data":"82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54"}
Nov 04 00:05:59.409812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.404169    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn" event={"ID":"bbe99feb-286e-41c3-98a4-d8f35d72eac9","Type":"ContainerDied","Data":"87b0d9d6ef4850ec263b074bf63c6a0e32e9f946ccb4471870c19380f32ab00f"}
Nov 04 00:05:59.409812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.404210    5074 scope.go:122] "RemoveContainer" containerID="82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54"
Nov 04 00:05:59.409812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.404374    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"
Nov 04 00:05:59.518211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.518185    5074 scope.go:122] "RemoveContainer" containerID="82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54"
Nov 04 00:05:59.519790 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:05:59.519311    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54\": container with ID starting with 82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54 not found: ID does not exist" containerID="82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54"
Nov 04 00:05:59.519790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.519348    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54"} err="failed to get container status \"82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54\": rpc error: code = NotFound desc = could not find container \"82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54\": container with ID starting with 82241ba5caebdc01e5aa237561b233666cc66e489fec9c5b0d4e7795ac6cfb54 not found: ID does not exist"
Nov 04 00:05:59.542167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.542136    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"]
Nov 04 00:05:59.563705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:05:59.563677    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-846dvwhwn"]
Nov 04 00:06:01.021529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:01.021494    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe99feb-286e-41c3-98a4-d8f35d72eac9" path="/var/lib/kubelet/pods/bbe99feb-286e-41c3-98a4-d8f35d72eac9/volumes"
Nov 04 00:06:08.764263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.764224    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"]
Nov 04 00:06:08.765150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765123    5074 cpu_manager.go:414] "containerMap: removing container" podUID="bbe99feb-286e-41c3-98a4-d8f35d72eac9" containerName="payload-processing"
Nov 04 00:06:08.765150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765144    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe99feb-286e-41c3-98a4-d8f35d72eac9" containerName="payload-processing"
Nov 04 00:06:08.765333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765163    5074 cpu_manager.go:414] "containerMap: removing container" podUID="59dcc78a-5607-40cd-8db0-d71206b4ad7d" containerName="revoke-keys"
Nov 04 00:06:08.765333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765168    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dcc78a-5607-40cd-8db0-d71206b4ad7d" containerName="revoke-keys"
Nov 04 00:06:08.765333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765200    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6636a4d9-8c65-49d3-a995-caf08906a24e" containerName="revoke-keys"
Nov 04 00:06:08.765333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765205    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6636a4d9-8c65-49d3-a995-caf08906a24e" containerName="revoke-keys"
Nov 04 00:06:08.765333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765317    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe99feb-286e-41c3-98a4-d8f35d72eac9" containerName="payload-processing"
Nov 04 00:06:08.765333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765329    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dcc78a-5607-40cd-8db0-d71206b4ad7d" containerName="revoke-keys"
Nov 04 00:06:08.765600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.765340    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6636a4d9-8c65-49d3-a995-caf08906a24e" containerName="revoke-keys"
Nov 04 00:06:08.770918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.770899    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.775558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.775529    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.775777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.775726    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.775868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.775806    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjwc\" (UniqueName: \"kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.781181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.781136    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"]
Nov 04 00:06:08.788011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.787949    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:06:08.788181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.787732    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:06:08.878060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.878012    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.878229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.878133    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.878229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.878176    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjwc\" (UniqueName: \"kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.878816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.878789    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.880536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.880509    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:08.959761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:08.959523    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjwc\" (UniqueName: \"kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc\") pod \"maas-api-revoke-sub-5bc99f5c-gnrlt\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:09.090924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:09.089231    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:09.393764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:09.393717    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"]
Nov 04 00:06:09.508175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:09.508139    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt" event={"ID":"c58eae36-21c7-45a0-a772-38e347a81bdb","Type":"ContainerStarted","Data":"33a64099a9fc6b934c45f148eef230cf0d0945c7c26ac8b0c734b798d6de179f"}
Nov 04 00:06:10.518223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:10.518176    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt" event={"ID":"c58eae36-21c7-45a0-a772-38e347a81bdb","Type":"ContainerStarted","Data":"d264372e55a9630f125cff3c6844a56f8aa263391e187de96d5d4a8c2b104fac"}
Nov 04 00:06:10.545353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:10.545296    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt" podStartSLOduration=2.545282044 podStartE2EDuration="2.545282044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:06:10.541421474 +0000 UTC m=+2235.641021596" watchObservedRunningTime="2026-11-04 00:06:10.545282044 +0000 UTC m=+2235.644882187"
Nov 04 00:06:11.445694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.445657    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"]
Nov 04 00:06:11.452490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.452462    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.463412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.463377    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"]
Nov 04 00:06:11.464449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.464397    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8895cb\""
Nov 04 00:06:11.464449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.464418    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-78dce14119dd\""
Nov 04 00:06:11.464666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.464524    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"openshift-service-ca.crt\""
Nov 04 00:06:11.467234 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.467213    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"routing-overlay\""
Nov 04 00:06:11.469900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.469872    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"kube-root-ca.crt\""
Nov 04 00:06:11.528423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.528386    5074 generic.go:356] "Generic (PLEG): container finished" podID="c58eae36-21c7-45a0-a772-38e347a81bdb" containerID="d264372e55a9630f125cff3c6844a56f8aa263391e187de96d5d4a8c2b104fac" exitCode=0
Nov 04 00:06:11.528856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.528472    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt" event={"ID":"c58eae36-21c7-45a0-a772-38e347a81bdb","Type":"ContainerDied","Data":"d264372e55a9630f125cff3c6844a56f8aa263391e187de96d5d4a8c2b104fac"}
Nov 04 00:06:11.554273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.554236    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.554446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.554388    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.554446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.554430    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.554532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.554516    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92j6w\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.657765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.655699    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92j6w\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.657765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.655989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.657765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.656125    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.657765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.656173    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.678762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.671112    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.678762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.671655    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.678762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.672144    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.731815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.718951    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92j6w\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:11.784935 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:11.777753    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:12.129783 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:06:12.129201    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20077c3e_eb59_4cf4_b413_1cafd649af8f.slice/crio-0640430e8473e1669c241a47dc4f230d0cc7f5ed013942142cbf8c27b8df8c0e WatchSource:0}: Error finding container 0640430e8473e1669c241a47dc4f230d0cc7f5ed013942142cbf8c27b8df8c0e: Status 404 returned error can't find the container with id 0640430e8473e1669c241a47dc4f230d0cc7f5ed013942142cbf8c27b8df8c0e
Nov 04 00:06:12.136406 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.136376    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"]
Nov 04 00:06:12.538385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.538331    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc" event={"ID":"20077c3e-eb59-4cf4-b413-1cafd649af8f","Type":"ContainerStarted","Data":"b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05"}
Nov 04 00:06:12.538830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.538439    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc" event={"ID":"20077c3e-eb59-4cf4-b413-1cafd649af8f","Type":"ContainerStarted","Data":"0640430e8473e1669c241a47dc4f230d0cc7f5ed013942142cbf8c27b8df8c0e"}
Nov 04 00:06:12.538830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.538472    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:12.575378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.575324    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc" podStartSLOduration=1.575311055 podStartE2EDuration="1.575311055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:06:12.571488416 +0000 UTC m=+2237.671088560" watchObservedRunningTime="2026-11-04 00:06:12.575311055 +0000 UTC m=+2237.674911199"
Nov 04 00:06:12.764625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.764589    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:12.879063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.878972    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca\") pod \"c58eae36-21c7-45a0-a772-38e347a81bdb\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") "
Nov 04 00:06:12.879282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.879234    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca\") pod \"c58eae36-21c7-45a0-a772-38e347a81bdb\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") "
Nov 04 00:06:12.879282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.879269    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc\" (UniqueName: \"kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc\") pod \"c58eae36-21c7-45a0-a772-38e347a81bdb\" (UID: \"c58eae36-21c7-45a0-a772-38e347a81bdb\") "
Nov 04 00:06:12.879770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.879722    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca" pod "c58eae36-21c7-45a0-a772-38e347a81bdb" (UID: "c58eae36-21c7-45a0-a772-38e347a81bdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:12.880014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.879982    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:12.882001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.881969    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca" pod "c58eae36-21c7-45a0-a772-38e347a81bdb" (UID: "c58eae36-21c7-45a0-a772-38e347a81bdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:06:12.882078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.882010    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc" pod "c58eae36-21c7-45a0-a772-38e347a81bdb" (UID: "c58eae36-21c7-45a0-a772-38e347a81bdb"). InnerVolumeSpecName "kube-api-access-8gjwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:12.981598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.981562    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjwc\" (UniqueName: \"kubernetes.io/projected/c58eae36-21c7-45a0-a772-38e347a81bdb-kube-api-access-8gjwc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:12.981598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:12.981598    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58eae36-21c7-45a0-a772-38e347a81bdb-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:13.548996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:13.548967    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"
Nov 04 00:06:13.549374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:13.549005    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt" event={"ID":"c58eae36-21c7-45a0-a772-38e347a81bdb","Type":"ContainerDied","Data":"33a64099a9fc6b934c45f148eef230cf0d0945c7c26ac8b0c734b798d6de179f"}
Nov 04 00:06:13.549374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:13.549055    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a64099a9fc6b934c45f148eef230cf0d0945c7c26ac8b0c734b798d6de179f"
Nov 04 00:06:17.007348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:17.007292    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"]
Nov 04 00:06:17.007907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:17.007615    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy" containerID="cri-o://67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f" gracePeriod=30
Nov 04 00:06:20.267344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:20.267310    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"]
Nov 04 00:06:20.971979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:20.971943    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"]
Nov 04 00:06:20.972174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:20.972148    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy" containerID="cri-o://d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c" gracePeriod=30
Nov 04 00:06:22.319372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.319322    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 04 00:06:22.387867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.387826    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.387880    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.387974    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388058    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388087    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388121    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388279    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.388357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388287    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:22.388357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388328    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388392    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket\") pod \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\" (UID: \"5b9db13d-1bd6-4f71-8a6a-d512e4e36970\") "
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388611    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388811    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.388895    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.389198    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.389218    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.389242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.389233    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.389713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.389262    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.389713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.389406    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:22.391257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.391228    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "kube-api-access-lkb2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:22.392269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.392243    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:06:22.395169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.395134    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:22.400163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.400131    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy" pod "5b9db13d-1bd6-4f71-8a6a-d512e4e36970" (UID: "5b9db13d-1bd6-4f71-8a6a-d512e4e36970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:22.490576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.490536    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkb2z\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-kube-api-access-lkb2z\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.490576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.490568    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.490576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.490580    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.490877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.490589    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.490877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.490598    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9db13d-1bd6-4f71-8a6a-d512e4e36970-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:22.678608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.678508    5074 generic.go:356] "Generic (PLEG): container finished" podID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerID="67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f" exitCode=0
Nov 04 00:06:22.678608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.678567    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"
Nov 04 00:06:22.678608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.678585    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" event={"ID":"5b9db13d-1bd6-4f71-8a6a-d512e4e36970","Type":"ContainerDied","Data":"67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f"}
Nov 04 00:06:22.678953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.678653    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49" event={"ID":"5b9db13d-1bd6-4f71-8a6a-d512e4e36970","Type":"ContainerDied","Data":"6261c4035eaba27bd07677d6d557e67f82d727cf058d09b65d485d4fb67a9e7e"}
Nov 04 00:06:22.678953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.678684    5074 scope.go:122] "RemoveContainer" containerID="67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f"
Nov 04 00:06:22.710635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.710605    5074 scope.go:122] "RemoveContainer" containerID="67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f"
Nov 04 00:06:22.712811 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:22.711200    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f\": container with ID starting with 67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f not found: ID does not exist" containerID="67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f"
Nov 04 00:06:22.712811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.711238    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f"} err="failed to get container status \"67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f\": rpc error: code = NotFound desc = could not find container \"67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f\": container with ID starting with 67fa6010c0de9206177e07973967d571c53332de5fb0d96bd2dc8b81c34dec5f not found: ID does not exist"
Nov 04 00:06:22.743076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.743036    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"]
Nov 04 00:06:22.748085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:22.748057    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-wqw49"]
Nov 04 00:06:23.020949 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:23.020916    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" path="/var/lib/kubelet/pods/5b9db13d-1bd6-4f71-8a6a-d512e4e36970/volumes"
Nov 04 00:06:23.453678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:23.453633    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"]
Nov 04 00:06:23.553422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:23.553390    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:23.689801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:23.689729    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy" containerID="cri-o://8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1" gracePeriod=30
Nov 04 00:06:24.076761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.076125    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"]
Nov 04 00:06:24.080800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.077467    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c58eae36-21c7-45a0-a772-38e347a81bdb" containerName="revoke-keys"
Nov 04 00:06:24.080800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.077495    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58eae36-21c7-45a0-a772-38e347a81bdb" containerName="revoke-keys"
Nov 04 00:06:24.080800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.077524    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy"
Nov 04 00:06:24.080800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.077532    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy"
Nov 04 00:06:24.080800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.077821    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58eae36-21c7-45a0-a772-38e347a81bdb" containerName="revoke-keys"
Nov 04 00:06:24.080800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.077837    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9db13d-1bd6-4f71-8a6a-d512e4e36970" containerName="istio-proxy"
Nov 04 00:06:24.093550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.084521    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.098041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.097616    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"e2e-cfl-140b39-provider-cred\""
Nov 04 00:06:24.106182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.103584    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"]
Nov 04 00:06:24.106917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.106867    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-e2e--9366bb2e-dockercfg-dqtxh\""
Nov 04 00:06:24.132701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.132652    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.132923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.132899    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcnl\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.132992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.132942    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.132992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.132979    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.234711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.234663    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcnl\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.234910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.234748    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.234910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.234800    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.235121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.235105    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.235694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.235663    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.235830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.235720    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.237579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.237552    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.248065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.248037    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcnl\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.448787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.448150    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:24.746122 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:06:24.746093    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d27399_d72f_4401_9a4f_760831884a99.slice/crio-7834e97936e0f43b4a857cdaa8460e5f96595a60653190bc2ec3f8f7032ab32e WatchSource:0}: Error finding container 7834e97936e0f43b4a857cdaa8460e5f96595a60653190bc2ec3f8f7032ab32e: Status 404 returned error can't find the container with id 7834e97936e0f43b4a857cdaa8460e5f96595a60653190bc2ec3f8f7032ab32e
Nov 04 00:06:24.748908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:24.748887    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"]
Nov 04 00:06:25.268810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.268772    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq" podUID="25d718c8-a83d-4b45-a1f5-c4fae9965f79" containerName="payload-processing" containerID="cri-o://5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2" gracePeriod=25
Nov 04 00:06:25.438904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.438867    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"]
Nov 04 00:06:25.439127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.439087    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy" containerID="cri-o://8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817" gracePeriod=30
Nov 04 00:06:25.647914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.647886    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.713777    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2" event={"ID":"d3d27399-d72f-4401-9a4f-760831884a99","Type":"ContainerStarted","Data":"d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb"}
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.713851    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2" event={"ID":"d3d27399-d72f-4401-9a4f-760831884a99","Type":"ContainerStarted","Data":"7834e97936e0f43b4a857cdaa8460e5f96595a60653190bc2ec3f8f7032ab32e"}
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.713910    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.716124    5074 generic.go:356] "Generic (PLEG): container finished" podID="25d718c8-a83d-4b45-a1f5-c4fae9965f79" containerID="5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2" exitCode=0
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.716274    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq" event={"ID":"25d718c8-a83d-4b45-a1f5-c4fae9965f79","Type":"ContainerDied","Data":"5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2"}
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.716319    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq" event={"ID":"25d718c8-a83d-4b45-a1f5-c4fae9965f79","Type":"ContainerDied","Data":"895e40aa6e5f03de6ba3a795a3e27a40c3815f1f20ea57fd8827edf087c26365"}
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.716350    5074 scope.go:122] "RemoveContainer" containerID="5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2"
Nov 04 00:06:25.718794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.716487    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"
Nov 04 00:06:25.749750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.749713    5074 scope.go:122] "RemoveContainer" containerID="5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2"
Nov 04 00:06:25.750166 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:25.750138    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2\": container with ID starting with 5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2 not found: ID does not exist" containerID="5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2"
Nov 04 00:06:25.750275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.750170    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2"} err="failed to get container status \"5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2\": rpc error: code = NotFound desc = could not find container \"5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2\": container with ID starting with 5ae6bc1d094e64c022d431a16fe74e9f386b2078b1065eed3e72c60210e33ab2 not found: ID does not exist"
Nov 04 00:06:25.754074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.754020    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2" podStartSLOduration=1.7540083800000001 podStartE2EDuration="1.75400838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:06:25.751029316 +0000 UTC m=+2250.850629459" watchObservedRunningTime="2026-11-04 00:06:25.75400838 +0000 UTC m=+2250.853608524"
Nov 04 00:06:25.774141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.774120    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials\") pod \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") "
Nov 04 00:06:25.774235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.774196    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt\") pod \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") "
Nov 04 00:06:25.774434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.774413    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume\") pod \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") "
Nov 04 00:06:25.774484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.774459    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume\") pod \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\" (UID: \"25d718c8-a83d-4b45-a1f5-c4fae9965f79\") "
Nov 04 00:06:25.774815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.774793    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume" pod "25d718c8-a83d-4b45-a1f5-c4fae9965f79" (UID: "25d718c8-a83d-4b45-a1f5-c4fae9965f79"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:25.774975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.774944    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume" pod "25d718c8-a83d-4b45-a1f5-c4fae9965f79" (UID: "25d718c8-a83d-4b45-a1f5-c4fae9965f79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:25.775275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.775254    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:25.775370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.775281    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d718c8-a83d-4b45-a1f5-c4fae9965f79-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:25.776454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.776432    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials" pod "25d718c8-a83d-4b45-a1f5-c4fae9965f79" (UID: "25d718c8-a83d-4b45-a1f5-c4fae9965f79"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:25.776545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.776528    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt" pod "25d718c8-a83d-4b45-a1f5-c4fae9965f79" (UID: "25d718c8-a83d-4b45-a1f5-c4fae9965f79"). InnerVolumeSpecName "kube-api-access-llkkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:25.876468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.876383    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:25.876468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:25.876418    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkkt\" (UniqueName: \"kubernetes.io/projected/25d718c8-a83d-4b45-a1f5-c4fae9965f79-kube-api-access-llkkt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.186229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.186189    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"]
Nov 04 00:06:26.196450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.196416    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dxnkfq"]
Nov 04 00:06:26.266658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.266632    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 04 00:06:26.383653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.383619    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.383868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.383690    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.383868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.383717    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.383868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.383796    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.383868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.383831    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.384115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.383953    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.384115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.384087    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.384239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.384191    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.384302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.384252    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy\") pod \"88df79fa-44f6-4a0d-8370-adae26d56a88\" (UID: \"88df79fa-44f6-4a0d-8370-adae26d56a88\") "
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.384083    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.384530    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.384939    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.385023    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.385046    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df79fa-44f6-4a0d-8370-adae26d56a88-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.385066    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.385111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.385084    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:26.386433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.386408    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:06:26.386535 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.386512    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "kube-api-access-8h9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:26.386634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.386615    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:26.386776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.386735    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy" pod "88df79fa-44f6-4a0d-8370-adae26d56a88" (UID: "88df79fa-44f6-4a0d-8370-adae26d56a88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:26.486582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486546    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.486582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486576    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.486582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486586    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.486851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486597    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.486851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486606    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h9lw\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-kube-api-access-8h9lw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.486851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486617    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df79fa-44f6-4a0d-8370-adae26d56a88-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.486851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.486625    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df79fa-44f6-4a0d-8370-adae26d56a88-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:26.728165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.728130    5074 generic.go:356] "Generic (PLEG): container finished" podID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerID="d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c" exitCode=0
Nov 04 00:06:26.728374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.728188    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"
Nov 04 00:06:26.728374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.728210    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" event={"ID":"88df79fa-44f6-4a0d-8370-adae26d56a88","Type":"ContainerDied","Data":"d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c"}
Nov 04 00:06:26.728374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.728264    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r" event={"ID":"88df79fa-44f6-4a0d-8370-adae26d56a88","Type":"ContainerDied","Data":"60f8afe73d6232ae4ccefb6eb8879667123c038f9c47908fcdd973adbde72d3f"}
Nov 04 00:06:26.728374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.728286    5074 scope.go:122] "RemoveContainer" containerID="d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c"
Nov 04 00:06:26.741358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.741339    5074 scope.go:122] "RemoveContainer" containerID="d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c"
Nov 04 00:06:26.741600 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:26.741582    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c\": container with ID starting with d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c not found: ID does not exist" containerID="d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c"
Nov 04 00:06:26.741650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.741618    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c"} err="failed to get container status \"d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c\": rpc error: code = NotFound desc = could not find container \"d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c\": container with ID starting with d2e1d71633ca9efe733b527e28064db329f19108b8ff15368ec4d601dd1e507c not found: ID does not exist"
Nov 04 00:06:26.769140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.769118    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"]
Nov 04 00:06:26.775817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:26.775796    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-d574r"]
Nov 04 00:06:27.022386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:27.022312    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d718c8-a83d-4b45-a1f5-c4fae9965f79" path="/var/lib/kubelet/pods/25d718c8-a83d-4b45-a1f5-c4fae9965f79/volumes"
Nov 04 00:06:27.023105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:27.023085    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" path="/var/lib/kubelet/pods/88df79fa-44f6-4a0d-8370-adae26d56a88/volumes"
Nov 04 00:06:28.025599 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:28.025552    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:06:28.978950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:28.978921    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:06:29.023029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.022990    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023065    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023110    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023155    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023270    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023348    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023428    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023472    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023623    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy\") pod \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\" (UID: \"1bb0e01c-2c82-419d-bc2d-d1d3477e607d\") "
Nov 04 00:06:29.023730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023701    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:29.023862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023836    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:29.023952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.023849    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:29.024205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.024178    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:29.024471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.024420    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:29.024610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.024476    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.024789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.024622    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.024789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.024640    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.024789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.024656    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.026332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.026303    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:29.026726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.026347    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "kube-api-access-wkwhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:29.026726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.026403    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:29.027307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.027280    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo" pod "1bb0e01c-2c82-419d-bc2d-d1d3477e607d" (UID: "1bb0e01c-2c82-419d-bc2d-d1d3477e607d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:06:29.129330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.129291    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.129330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.129321    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.129330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.129338    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.129700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.129347    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.129700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.129358    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwhv\" (UniqueName: \"kubernetes.io/projected/1bb0e01c-2c82-419d-bc2d-d1d3477e607d-kube-api-access-wkwhv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:29.757064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.757023    5074 generic.go:356] "Generic (PLEG): container finished" podID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerID="8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1" exitCode=0
Nov 04 00:06:29.757254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.757069    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"
Nov 04 00:06:29.757254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.757099    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" event={"ID":"1bb0e01c-2c82-419d-bc2d-d1d3477e607d","Type":"ContainerDied","Data":"8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1"}
Nov 04 00:06:29.757254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.757148    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr" event={"ID":"1bb0e01c-2c82-419d-bc2d-d1d3477e607d","Type":"ContainerDied","Data":"e59f1deb3f461998aa257b556b65e39c46d54fa39d8933799a325e6822eabbd7"}
Nov 04 00:06:29.757254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.757169    5074 scope.go:122] "RemoveContainer" containerID="8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1"
Nov 04 00:06:29.768911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.768891    5074 scope.go:122] "RemoveContainer" containerID="8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1"
Nov 04 00:06:29.769184 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:29.769163    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1\": container with ID starting with 8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1 not found: ID does not exist" containerID="8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1"
Nov 04 00:06:29.769259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.769192    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1"} err="failed to get container status \"8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1\": rpc error: code = NotFound desc = could not find container \"8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1\": container with ID starting with 8985ad28f9f22dda675703b7c599931dd537e7e8979b13a82bc1ee8f772b2cc1 not found: ID does not exist"
Nov 04 00:06:29.800257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.800232    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"]
Nov 04 00:06:29.809369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:29.809349    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghqfr"]
Nov 04 00:06:30.729287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.729260    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 04 00:06:30.765898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.765859    5074 generic.go:356] "Generic (PLEG): container finished" podID="21434b03-716d-4849-b8c3-490b5f807b51" containerID="8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817" exitCode=0
Nov 04 00:06:30.766076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.765917    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"
Nov 04 00:06:30.766076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.765950    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" event={"ID":"21434b03-716d-4849-b8c3-490b5f807b51","Type":"ContainerDied","Data":"8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817"}
Nov 04 00:06:30.766076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.766035    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb" event={"ID":"21434b03-716d-4849-b8c3-490b5f807b51","Type":"ContainerDied","Data":"ccde984e54eab29ec37a97fa7efce2689769fc98294b93b3a6650554b7bd7186"}
Nov 04 00:06:30.766076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.766068    5074 scope.go:122] "RemoveContainer" containerID="8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817"
Nov 04 00:06:30.780065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.780042    5074 scope.go:122] "RemoveContainer" containerID="8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817"
Nov 04 00:06:30.780363 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:30.780340    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817\": container with ID starting with 8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817 not found: ID does not exist" containerID="8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817"
Nov 04 00:06:30.780445 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.780374    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817"} err="failed to get container status \"8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817\": rpc error: code = NotFound desc = could not find container \"8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817\": container with ID starting with 8f5a1f4fec522f15dcf28e1a9957ae2845a7beafe1484e9f75bbc92a461a2817 not found: ID does not exist"
Nov 04 00:06:30.858058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858027    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858104    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858162    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858190    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858241    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858325    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858361    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858454    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858511    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket\") pod \"21434b03-716d-4849-b8c3-490b5f807b51\" (UID: \"21434b03-716d-4849-b8c3-490b5f807b51\") "
Nov 04 00:06:30.858579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858547    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:30.858579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858561    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:30.858857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858672    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:30.858857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.858832    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:30.859040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.859012    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:30.859243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.859224    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.859315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.859249    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.859315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.859264    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21434b03-716d-4849-b8c3-490b5f807b51-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.859315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.859279    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.859315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.859293    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.860653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.860625    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "kube-api-access-c9mjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:30.860928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.860895    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:30.861128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.861105    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:30.861367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.861348    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo" pod "21434b03-716d-4849-b8c3-490b5f807b51" (UID: "21434b03-716d-4849-b8c3-490b5f807b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:06:30.960645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.960574    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mjh\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-kube-api-access-c9mjh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.960645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.960602    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21434b03-716d-4849-b8c3-490b5f807b51-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.960645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.960614    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21434b03-716d-4849-b8c3-490b5f807b51-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:30.960645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:30.960623    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21434b03-716d-4849-b8c3-490b5f807b51-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:31.020188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:31.020149    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" path="/var/lib/kubelet/pods/1bb0e01c-2c82-419d-bc2d-d1d3477e607d/volumes"
Nov 04 00:06:31.097817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:31.097778    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"]
Nov 04 00:06:31.103460 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:31.103431    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-wrrgb"]
Nov 04 00:06:33.020866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:33.020820    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21434b03-716d-4849-b8c3-490b5f807b51" path="/var/lib/kubelet/pods/21434b03-716d-4849-b8c3-490b5f807b51/volumes"
Nov 04 00:06:36.734157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:36.734123    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:06:36.802532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:36.802495    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"]
Nov 04 00:06:41.803111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:41.803069    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc" podUID="20077c3e-eb59-4cf4-b413-1cafd649af8f" containerName="payload-processing" containerID="cri-o://b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05" gracePeriod=25
Nov 04 00:06:42.088683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.088658    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:42.130812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.130780    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w\") pod \"20077c3e-eb59-4cf4-b413-1cafd649af8f\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") "
Nov 04 00:06:42.131000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.130859    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials\") pod \"20077c3e-eb59-4cf4-b413-1cafd649af8f\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") "
Nov 04 00:06:42.131075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.131013    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume\") pod \"20077c3e-eb59-4cf4-b413-1cafd649af8f\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") "
Nov 04 00:06:42.131131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.131102    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume\") pod \"20077c3e-eb59-4cf4-b413-1cafd649af8f\" (UID: \"20077c3e-eb59-4cf4-b413-1cafd649af8f\") "
Nov 04 00:06:42.131935 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.131905    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume" pod "20077c3e-eb59-4cf4-b413-1cafd649af8f" (UID: "20077c3e-eb59-4cf4-b413-1cafd649af8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:42.132322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.132295    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume" pod "20077c3e-eb59-4cf4-b413-1cafd649af8f" (UID: "20077c3e-eb59-4cf4-b413-1cafd649af8f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:42.135218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.135192    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials" pod "20077c3e-eb59-4cf4-b413-1cafd649af8f" (UID: "20077c3e-eb59-4cf4-b413-1cafd649af8f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:42.144175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.144153    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w" pod "20077c3e-eb59-4cf4-b413-1cafd649af8f" (UID: "20077c3e-eb59-4cf4-b413-1cafd649af8f"). InnerVolumeSpecName "kube-api-access-92j6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:42.232703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.232666    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92j6w\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-kube-api-access-92j6w\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:42.232703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.232698    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20077c3e-eb59-4cf4-b413-1cafd649af8f-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:42.232703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.232710    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:42.232970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.232719    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20077c3e-eb59-4cf4-b413-1cafd649af8f-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:42.883996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.883958    5074 generic.go:356] "Generic (PLEG): container finished" podID="20077c3e-eb59-4cf4-b413-1cafd649af8f" containerID="b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05" exitCode=0
Nov 04 00:06:42.884499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.884011    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"
Nov 04 00:06:42.884499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.884039    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc" event={"ID":"20077c3e-eb59-4cf4-b413-1cafd649af8f","Type":"ContainerDied","Data":"b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05"}
Nov 04 00:06:42.884499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.884095    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc" event={"ID":"20077c3e-eb59-4cf4-b413-1cafd649af8f","Type":"ContainerDied","Data":"0640430e8473e1669c241a47dc4f230d0cc7f5ed013942142cbf8c27b8df8c0e"}
Nov 04 00:06:42.884499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.884118    5074 scope.go:122] "RemoveContainer" containerID="b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05"
Nov 04 00:06:42.895637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.895620    5074 scope.go:122] "RemoveContainer" containerID="b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05"
Nov 04 00:06:42.895984 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:42.895956    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05\": container with ID starting with b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05 not found: ID does not exist" containerID="b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05"
Nov 04 00:06:42.896107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.895994    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05"} err="failed to get container status \"b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05\": rpc error: code = NotFound desc = could not find container \"b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05\": container with ID starting with b4eddd5eddbfd714e18999c9132ae880385b207e51badbed077b8fc57cb59e05 not found: ID does not exist"
Nov 04 00:06:42.919477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.919447    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"]
Nov 04 00:06:42.927179 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:42.927155    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b9bsdc"]
Nov 04 00:06:43.019634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:43.019600    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20077c3e-eb59-4cf4-b413-1cafd649af8f" path="/var/lib/kubelet/pods/20077c3e-eb59-4cf4-b413-1cafd649af8f/volumes"
Nov 04 00:06:50.269529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.269496    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"]
Nov 04 00:06:50.270665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270631    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy"
Nov 04 00:06:50.270665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270661    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270693    5074 cpu_manager.go:414] "containerMap: removing container" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270701    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270734    5074 cpu_manager.go:414] "containerMap: removing container" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270759    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270785    5074 cpu_manager.go:414] "containerMap: removing container" podUID="20077c3e-eb59-4cf4-b413-1cafd649af8f" containerName="payload-processing"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270793    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20077c3e-eb59-4cf4-b413-1cafd649af8f" containerName="payload-processing"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270842    5074 cpu_manager.go:414] "containerMap: removing container" podUID="25d718c8-a83d-4b45-a1f5-c4fae9965f79" containerName="payload-processing"
Nov 04 00:06:50.270936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.270850    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d718c8-a83d-4b45-a1f5-c4fae9965f79" containerName="payload-processing"
Nov 04 00:06:50.271353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.271034    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="20077c3e-eb59-4cf4-b413-1cafd649af8f" containerName="payload-processing"
Nov 04 00:06:50.271353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.271057    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d718c8-a83d-4b45-a1f5-c4fae9965f79" containerName="payload-processing"
Nov 04 00:06:50.271353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.271073    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb0e01c-2c82-419d-bc2d-d1d3477e607d" containerName="istio-proxy"
Nov 04 00:06:50.271353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.271086    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="88df79fa-44f6-4a0d-8370-adae26d56a88" containerName="istio-proxy"
Nov 04 00:06:50.271353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.271100    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="21434b03-716d-4849-b8c3-490b5f807b51" containerName="istio-proxy"
Nov 04 00:06:50.281474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.281448    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"]
Nov 04 00:06:50.281691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.281610    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.293454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.293429    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:06:50.296191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.296171    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:06:50.385605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.385566    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.385813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.385716    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.385889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.385829    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlxs\" (UniqueName: \"kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.487611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.487577    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.487822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.487637    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlxs\" (UniqueName: \"kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.487883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.487831    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.488484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.488454    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.489988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.489967    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.501071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.501051    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlxs\" (UniqueName: \"kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs\") pod \"maas-api-revoke-sub-da72753f-vl86x\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.613069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.612977    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:50.897643 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:06:50.897613    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741f943e_9f7e_4fb1_bbe5_c3acf2e9544e.slice/crio-6b1b78202c510ab9677a40c4f1cf1aed535aa96dfea5bba2ba788a4b85637b3b WatchSource:0}: Error finding container 6b1b78202c510ab9677a40c4f1cf1aed535aa96dfea5bba2ba788a4b85637b3b: Status 404 returned error can't find the container with id 6b1b78202c510ab9677a40c4f1cf1aed535aa96dfea5bba2ba788a4b85637b3b
Nov 04 00:06:50.899120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.899089    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"]
Nov 04 00:06:50.996712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:50.996680    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x" event={"ID":"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e","Type":"ContainerStarted","Data":"6b1b78202c510ab9677a40c4f1cf1aed535aa96dfea5bba2ba788a4b85637b3b"}
Nov 04 00:06:52.007039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:52.007003    5074 generic.go:356] "Generic (PLEG): container finished" podID="741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" containerID="ed35d696b73092788506dfeeedc1347d1e251e3efa67ed4428ce7a6e29920e08" exitCode=0
Nov 04 00:06:52.007459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:52.007383    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x" event={"ID":"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e","Type":"ContainerDied","Data":"ed35d696b73092788506dfeeedc1347d1e251e3efa67ed4428ce7a6e29920e08"}
Nov 04 00:06:53.208688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.208661    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:53.330659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.330577    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca\") pod \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") "
Nov 04 00:06:53.330898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.330663    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs\" (UniqueName: \"kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs\") pod \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") "
Nov 04 00:06:53.330898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.330777    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca\") pod \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\" (UID: \"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e\") "
Nov 04 00:06:53.331055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.330970    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca" pod "741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" (UID: "741f943e-9f7e-4fb1-bbe5-c3acf2e9544e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:53.331263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.331239    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:53.332895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.332861    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca" pod "741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" (UID: "741f943e-9f7e-4fb1-bbe5-c3acf2e9544e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:06:53.333024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.332993    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs" pod "741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" (UID: "741f943e-9f7e-4fb1-bbe5-c3acf2e9544e"). InnerVolumeSpecName "kube-api-access-9xlxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:53.432445 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.432408    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlxs\" (UniqueName: \"kubernetes.io/projected/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-kube-api-access-9xlxs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:53.432445 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:53.432440    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:54.055274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:54.053558    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x" event={"ID":"741f943e-9f7e-4fb1-bbe5-c3acf2e9544e","Type":"ContainerDied","Data":"6b1b78202c510ab9677a40c4f1cf1aed535aa96dfea5bba2ba788a4b85637b3b"}
Nov 04 00:06:54.055274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:54.053607    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"
Nov 04 00:06:54.055610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:54.053621    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1b78202c510ab9677a40c4f1cf1aed535aa96dfea5bba2ba788a4b85637b3b"
Nov 04 00:06:58.022808 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:06:58.022734    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:06:58.960984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:58.960949    5074 scope.go:122] "RemoveContainer" containerID="b315bed25e05991485f7a66d819f2cd651c99050ee8607b67c7d23f3156ebfd5"
Nov 04 00:06:58.971713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:58.971689    5074 scope.go:122] "RemoveContainer" containerID="594c2e6e84af8f7dd9ea6655b5779feb95116afb6b70ff59d6c225173f2fcaf4"
Nov 04 00:06:58.984912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:58.984868    5074 scope.go:122] "RemoveContainer" containerID="4b6279783e4d407e930f62ea2553490efeeaa1d3f662e6da1b22bf5e3fc686eb"
Nov 04 00:06:58.994563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:06:58.994540    5074 scope.go:122] "RemoveContainer" containerID="63397aca03160066fccdd59859470886382e857944fe8926d134b80c71129856"
Nov 04 00:07:01.756842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.756801    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"]
Nov 04 00:07:01.780078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.758852    5074 cpu_manager.go:414] "containerMap: removing container" podUID="741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" containerName="revoke-keys"
Nov 04 00:07:01.780340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.780313    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" containerName="revoke-keys"
Nov 04 00:07:01.780860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.780833    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" containerName="revoke-keys"
Nov 04 00:07:01.868826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.832311    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"]
Nov 04 00:07:01.868826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.832448    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:01.877203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.873987    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-d87e354e0f00\""
Nov 04 00:07:01.877203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.874135    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\""
Nov 04 00:07:01.877203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.874230    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\""
Nov 04 00:07:01.877203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.876960    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\""
Nov 04 00:07:01.877203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.877093    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-261bd6139428\""
Nov 04 00:07:01.877203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.877164    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\""
Nov 04 00:07:01.878691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.878670    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-hsvcw\""
Nov 04 00:07:01.917941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.917899    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:01.918111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.918036    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blnt\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:01.918282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.918251    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:01.918385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:01.918357    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.019530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.019438    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.019530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.019521    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blnt\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.019731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.019638    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.019731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.019670    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.020467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.020436    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.020583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.020450    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.022548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.022523    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.033508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.033416    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blnt\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.195358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.195321    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:02.221637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.221593    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"]
Nov 04 00:07:02.260875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.228801    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.262129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.261730    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:07:02.262129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.261839    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:07:02.271013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.270949    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"]
Nov 04 00:07:02.337760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.333354    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.337760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.333446    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmtg\" (UniqueName: \"kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.337760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.333637    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.462924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.456107    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.462924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.456193    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmtg\" (UniqueName: \"kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.462924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.456354    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.462924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.457730    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.487036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.486999    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.561467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.561398    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmtg\" (UniqueName: \"kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg\") pod \"maas-api-revoke-sub-db7a465d-877b8\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.653029 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:02.653000    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccae4a8_1d52_43d1_a4c4_bd6400da574a.slice/crio-f3f515c20431454b4ee93b332c0bf22c0644a7dcb8efc056fb3b477dd88e355f WatchSource:0}: Error finding container f3f515c20431454b4ee93b332c0bf22c0644a7dcb8efc056fb3b477dd88e355f: Status 404 returned error can't find the container with id f3f515c20431454b4ee93b332c0bf22c0644a7dcb8efc056fb3b477dd88e355f
Nov 04 00:07:02.656265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.656243    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"]
Nov 04 00:07:02.670700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.670676    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:02.970171 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:02.970140    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9a2a3b_077b_4b5a_9e4f_1b6cd1409606.slice/crio-e933629614dcf47a408f42a6657e871e6b948bb1d6359c19ee3e13cb069e6f35 WatchSource:0}: Error finding container e933629614dcf47a408f42a6657e871e6b948bb1d6359c19ee3e13cb069e6f35: Status 404 returned error can't find the container with id e933629614dcf47a408f42a6657e871e6b948bb1d6359c19ee3e13cb069e6f35
Nov 04 00:07:02.973909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:02.973886    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"]
Nov 04 00:07:03.147479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:03.147437    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr" event={"ID":"fccae4a8-1d52-43d1-a4c4-bd6400da574a","Type":"ContainerStarted","Data":"f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86"}
Nov 04 00:07:03.147696 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:03.147515    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:03.147696 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:03.147541    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr" event={"ID":"fccae4a8-1d52-43d1-a4c4-bd6400da574a","Type":"ContainerStarted","Data":"f3f515c20431454b4ee93b332c0bf22c0644a7dcb8efc056fb3b477dd88e355f"}
Nov 04 00:07:03.149000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:03.148973    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8" event={"ID":"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606","Type":"ContainerStarted","Data":"e933629614dcf47a408f42a6657e871e6b948bb1d6359c19ee3e13cb069e6f35"}
Nov 04 00:07:03.194603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:03.194553    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr" podStartSLOduration=2.194538971 podStartE2EDuration="2.194538971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:03.185034982 +0000 UTC m=+2288.284635125" watchObservedRunningTime="2026-11-04 00:07:03.194538971 +0000 UTC m=+2288.294139114"
Nov 04 00:07:04.157820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:04.157783    5074 generic.go:356] "Generic (PLEG): container finished" podID="ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" containerID="52cfd6570478c2a23506d5276ac61c8f43c28d0c48124c3aa39229e1bc161784" exitCode=0
Nov 04 00:07:04.158318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:04.157865    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8" event={"ID":"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606","Type":"ContainerDied","Data":"52cfd6570478c2a23506d5276ac61c8f43c28d0c48124c3aa39229e1bc161784"}
Nov 04 00:07:05.296669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.296629    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"]
Nov 04 00:07:05.297324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.296895    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy" containerID="cri-o://842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116" gracePeriod=30
Nov 04 00:07:05.358962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.358932    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:05.413718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.401251    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca\") pod \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") "
Nov 04 00:07:05.413718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.401389    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca\") pod \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") "
Nov 04 00:07:05.413718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.401453    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg\" (UniqueName: \"kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg\") pod \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\" (UID: \"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606\") "
Nov 04 00:07:05.413718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.406709    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca" pod "ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" (UID: "ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:05.413718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.412435    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca" pod "ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" (UID: "ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:07:05.419868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.417476    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg" pod "ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" (UID: "ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606"). InnerVolumeSpecName "kube-api-access-8xmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:05.503426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.503386    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:05.503426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.503421    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmtg\" (UniqueName: \"kubernetes.io/projected/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-kube-api-access-8xmtg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:05.503426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:05.503433    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:06.174518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.174474    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"
Nov 04 00:07:06.174518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.174497    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8" event={"ID":"ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606","Type":"ContainerDied","Data":"e933629614dcf47a408f42a6657e871e6b948bb1d6359c19ee3e13cb069e6f35"}
Nov 04 00:07:06.174815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.174558    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e933629614dcf47a408f42a6657e871e6b948bb1d6359c19ee3e13cb069e6f35"
Nov 04 00:07:06.589760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.588946    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"]
Nov 04 00:07:06.590287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.590215    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" containerName="revoke-keys"
Nov 04 00:07:06.590287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.590238    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" containerName="revoke-keys"
Nov 04 00:07:06.600141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.590462    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" containerName="revoke-keys"
Nov 04 00:07:06.613293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.603113    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"]
Nov 04 00:07:06.613293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.603233    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.620393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.618424    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"payload-processing-external-model-e2e--4ce224e7-dockercfg-9h7nn\""
Nov 04 00:07:06.620393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.618482    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"e2e-cmp-legacy-4c1ffa-provider-cred\""
Nov 04 00:07:06.621070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.621040    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"openshift-service-ca.crt\""
Nov 04 00:07:06.632397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.632370    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"routing-overlay\""
Nov 04 00:07:06.635982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.635952    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-fa0d0a\""
Nov 04 00:07:06.636113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.636063    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"kube-root-ca.crt\""
Nov 04 00:07:06.734761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.731182    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cx6\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.734761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.731311    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.734761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.731525    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.734761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.731995    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.840825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.840006    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.840825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.840113    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cx6\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.840825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.840178    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.840825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.840302    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.844760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.843158    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.854820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.852053    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.859345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.859315    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.870490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.866485    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cx6\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:06.925532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:06.925500    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:07.219809 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:07.219771    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fab3cd_b25a_44bc_b85d_30efb373a0b6.slice/crio-84cc8b0c314078193c10510ea864c41bd2841ce5ae44ff234cd21833cc4590bd WatchSource:0}: Error finding container 84cc8b0c314078193c10510ea864c41bd2841ce5ae44ff234cd21833cc4590bd: Status 404 returned error can't find the container with id 84cc8b0c314078193c10510ea864c41bd2841ce5ae44ff234cd21833cc4590bd
Nov 04 00:07:07.224031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:07.224005    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"]
Nov 04 00:07:08.193068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:08.193030    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5" event={"ID":"92fab3cd-b25a-44bc-b85d-30efb373a0b6","Type":"ContainerStarted","Data":"4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5"}
Nov 04 00:07:08.193556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:08.193094    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5" event={"ID":"92fab3cd-b25a-44bc-b85d-30efb373a0b6","Type":"ContainerStarted","Data":"84cc8b0c314078193c10510ea864c41bd2841ce5ae44ff234cd21833cc4590bd"}
Nov 04 00:07:08.193556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:08.193173    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:08.228391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:08.228341    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5" podStartSLOduration=2.22832689 podStartE2EDuration="2.22832689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:08.228075271 +0000 UTC m=+2293.327675416" watchObservedRunningTime="2026-11-04 00:07:08.22832689 +0000 UTC m=+2293.327927034"
Nov 04 00:07:08.923127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:08.922367    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"]
Nov 04 00:07:08.923127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:08.922622    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy" containerID="cri-o://d477700ee33b9c250ab34bab6400b2cc8f5cf5215c159a1721db207cd1e27f39" gracePeriod=30
Nov 04 00:07:11.014319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.014281    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 04 00:07:11.127067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.126974    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127014    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127045    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127077    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127103    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127164    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127208    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127370    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127410    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket\") pod \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\" (UID: \"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f\") "
Nov 04 00:07:11.127596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127536    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:11.127596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127554    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:11.127961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.127936    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:11.128050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.128022    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:11.128187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.128165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:11.128255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.128190    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.128255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.128213    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.128255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.128227    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.128255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.128241    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.129545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.129518    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:11.130037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.129999    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:11.130331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.130305    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "kube-api-access-87shf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:11.130516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.130488    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo" pod "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" (UID: "10d1dc37-2703-49d4-852a-b0d3cc3cdc3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:07:11.219439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.219400    5074 generic.go:356] "Generic (PLEG): container finished" podID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerID="842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116" exitCode=0
Nov 04 00:07:11.219637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.219457    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"
Nov 04 00:07:11.219637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.219489    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" event={"ID":"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f","Type":"ContainerDied","Data":"842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116"}
Nov 04 00:07:11.219637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.219567    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r" event={"ID":"10d1dc37-2703-49d4-852a-b0d3cc3cdc3f","Type":"ContainerDied","Data":"a0ea4482b00759847cddc6aad480484762c83d2d7bd4fe8c6cb4834ba1dfb666"}
Nov 04 00:07:11.219637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.219597    5074 scope.go:122] "RemoveContainer" containerID="842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116"
Nov 04 00:07:11.229551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.229525    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.229660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.229552    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87shf\" (UniqueName: \"kubernetes.io/projected/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-kube-api-access-87shf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.229660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.229569    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.229660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.229583    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.229660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.229596    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:11.231974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.231955    5074 scope.go:122] "RemoveContainer" containerID="842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116"
Nov 04 00:07:11.232540 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:07:11.232520    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116\": container with ID starting with 842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116 not found: ID does not exist" containerID="842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116"
Nov 04 00:07:11.232615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.232554    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116"} err="failed to get container status \"842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116\": rpc error: code = NotFound desc = could not find container \"842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116\": container with ID starting with 842adebd6fc13cfd854f6443209e5a1fad7b4193218832605b3d8b97720d9116 not found: ID does not exist"
Nov 04 00:07:11.259048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.259019    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"]
Nov 04 00:07:11.280639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:11.280605    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-fmp4r"]
Nov 04 00:07:13.021002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.020965    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" path="/var/lib/kubelet/pods/10d1dc37-2703-49d4-852a-b0d3cc3cdc3f/volumes"
Nov 04 00:07:13.892963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.892929    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"]
Nov 04 00:07:13.896861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.896826    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:13.939632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.939602    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"]
Nov 04 00:07:13.940721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.940698    5074 cpu_manager.go:414] "containerMap: removing container" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy"
Nov 04 00:07:13.941006 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.940985    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy"
Nov 04 00:07:13.941418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.941399    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d1dc37-2703-49d4-852a-b0d3cc3cdc3f" containerName="istio-proxy"
Nov 04 00:07:13.945581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.945559    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:13.948801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:13.948778    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"]
Nov 04 00:07:14.081627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.081582    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.082330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.082299    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtcl\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.082584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.082563    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.082783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.082766    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.191941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.191642    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtcl\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.192536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.192513    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.193362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.193297    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.193519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.193503    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.194777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.194760    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.195499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.195390    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.200544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.200445    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.214598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.214567    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtcl\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.267546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.267517    5074 generic.go:356] "Generic (PLEG): container finished" podID="975814af-8159-4623-b09e-44c15fb2ddc1" containerID="d477700ee33b9c250ab34bab6400b2cc8f5cf5215c159a1721db207cd1e27f39" exitCode=0
Nov 04 00:07:14.267835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.267588    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" event={"ID":"975814af-8159-4623-b09e-44c15fb2ddc1","Type":"ContainerDied","Data":"d477700ee33b9c250ab34bab6400b2cc8f5cf5215c159a1721db207cd1e27f39"}
Nov 04 00:07:14.321948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.321922    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:14.364725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.364689    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:07:14.398430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398376    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398429    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398451    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398513    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398588    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398630    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398655    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398711    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.398789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.398768    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket\") pod \"975814af-8159-4623-b09e-44c15fb2ddc1\" (UID: \"975814af-8159-4623-b09e-44c15fb2ddc1\") "
Nov 04 00:07:14.399557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.399524    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:14.400490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.400465    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:14.402847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.402822    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:14.402962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.402880    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:14.403138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.403120    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:14.406945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.406904    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:07:14.406945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.406911    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:14.429211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.429176    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "kube-api-access-mz4xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:14.429938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.429899    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token" pod "975814af-8159-4623-b09e-44c15fb2ddc1" (UID: "975814af-8159-4623-b09e-44c15fb2ddc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:14.483831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.479813    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"]
Nov 04 00:07:14.483831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.481314    5074 cpu_manager.go:414] "containerMap: removing container" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy"
Nov 04 00:07:14.483831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.481336    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy"
Nov 04 00:07:14.483831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.481520    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" containerName="istio-proxy"
Nov 04 00:07:14.486317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.486295    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"]
Nov 04 00:07:14.486450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.486393    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.491261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.491235    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:07:14.494181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.494160    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:07:14.500910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.500853    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.501040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.500960    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.501099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501066    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qchdr\" (UniqueName: \"kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.501286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501270    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501289    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501300    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501310    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975814af-8159-4623-b09e-44c15fb2ddc1-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501318    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501329    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501338    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975814af-8159-4623-b09e-44c15fb2ddc1-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501348    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975814af-8159-4623-b09e-44c15fb2ddc1-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.501452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.501357    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4xd\" (UniqueName: \"kubernetes.io/projected/975814af-8159-4623-b09e-44c15fb2ddc1-kube-api-access-mz4xd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:14.602236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.602108    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.602439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.602292    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.602439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.602389    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qchdr\" (UniqueName: \"kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.603259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.603236    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.605170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.605142    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.617917 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:14.617890    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d26fce_0711_43f0_96de_97670163c914.slice/crio-82c88b290f77781af918ee4579e65eb175c4ea3f592ce32e71b8b78f7c763a8d WatchSource:0}: Error finding container 82c88b290f77781af918ee4579e65eb175c4ea3f592ce32e71b8b78f7c763a8d: Status 404 returned error can't find the container with id 82c88b290f77781af918ee4579e65eb175c4ea3f592ce32e71b8b78f7c763a8d
Nov 04 00:07:14.621389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.621326    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qchdr\" (UniqueName: \"kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr\") pod \"maas-api-revoke-sub-9c34bc3e-tqkgh\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:14.621839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.621813    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"]
Nov 04 00:07:14.803456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:14.803378    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:15.089799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.089762    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"]
Nov 04 00:07:15.090200 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:15.090170    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5f4282_4d1a_467e_b3c5_e0c361968222.slice/crio-ae74facffce08a3954807ddb6f2ea134c2e40e5eb7b400b2479c8130f942cd31 WatchSource:0}: Error finding container ae74facffce08a3954807ddb6f2ea134c2e40e5eb7b400b2479c8130f942cd31: Status 404 returned error can't find the container with id ae74facffce08a3954807ddb6f2ea134c2e40e5eb7b400b2479c8130f942cd31
Nov 04 00:07:15.277420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.277381    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6" event={"ID":"e6d26fce-0711-43f0-96de-97670163c914","Type":"ContainerStarted","Data":"351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb"}
Nov 04 00:07:15.277629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.277447    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6" event={"ID":"e6d26fce-0711-43f0-96de-97670163c914","Type":"ContainerStarted","Data":"82c88b290f77781af918ee4579e65eb175c4ea3f592ce32e71b8b78f7c763a8d"}
Nov 04 00:07:15.277629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.277472    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:15.278878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.278851    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh" event={"ID":"fa5f4282-4d1a-467e-b3c5-e0c361968222","Type":"ContainerStarted","Data":"ae74facffce08a3954807ddb6f2ea134c2e40e5eb7b400b2479c8130f942cd31"}
Nov 04 00:07:15.280322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.280289    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"
Nov 04 00:07:15.280438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.280314    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6" event={"ID":"975814af-8159-4623-b09e-44c15fb2ddc1","Type":"ContainerDied","Data":"432d79b0359f011dc2daff9cbe58be5458d51c4c842045adb811e15f0f868942"}
Nov 04 00:07:15.280438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.280377    5074 scope.go:122] "RemoveContainer" containerID="d477700ee33b9c250ab34bab6400b2cc8f5cf5215c159a1721db207cd1e27f39"
Nov 04 00:07:15.326574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.326521    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6" podStartSLOduration=2.326506686 podStartE2EDuration="2.326506686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:15.322874484 +0000 UTC m=+2300.422474630" watchObservedRunningTime="2026-11-04 00:07:15.326506686 +0000 UTC m=+2300.426106829"
Nov 04 00:07:15.396583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.396506    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"]
Nov 04 00:07:15.401816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:15.401786    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-twmb6"]
Nov 04 00:07:16.290281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:16.290188    5074 generic.go:356] "Generic (PLEG): container finished" podID="fa5f4282-4d1a-467e-b3c5-e0c361968222" containerID="5eda02dcb5c33d3a9a8d3e7d498c59761ba5ca0f85f0cb6ef4a502657d8e18f8" exitCode=0
Nov 04 00:07:16.290281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:16.290231    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh" event={"ID":"fa5f4282-4d1a-467e-b3c5-e0c361968222","Type":"ContainerDied","Data":"5eda02dcb5c33d3a9a8d3e7d498c59761ba5ca0f85f0cb6ef4a502657d8e18f8"}
Nov 04 00:07:17.020806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.020717    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975814af-8159-4623-b09e-44c15fb2ddc1" path="/var/lib/kubelet/pods/975814af-8159-4623-b09e-44c15fb2ddc1/volumes"
Nov 04 00:07:17.504188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.504154    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:17.546442 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.546397    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr\" (UniqueName: \"kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr\") pod \"fa5f4282-4d1a-467e-b3c5-e0c361968222\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") "
Nov 04 00:07:17.546678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.546620    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca\") pod \"fa5f4282-4d1a-467e-b3c5-e0c361968222\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") "
Nov 04 00:07:17.546797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.546715    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca\") pod \"fa5f4282-4d1a-467e-b3c5-e0c361968222\" (UID: \"fa5f4282-4d1a-467e-b3c5-e0c361968222\") "
Nov 04 00:07:17.548158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.548122    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca" pod "fa5f4282-4d1a-467e-b3c5-e0c361968222" (UID: "fa5f4282-4d1a-467e-b3c5-e0c361968222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:17.550410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.550330    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca" pod "fa5f4282-4d1a-467e-b3c5-e0c361968222" (UID: "fa5f4282-4d1a-467e-b3c5-e0c361968222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:07:17.550863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.550835    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr" pod "fa5f4282-4d1a-467e-b3c5-e0c361968222" (UID: "fa5f4282-4d1a-467e-b3c5-e0c361968222"). InnerVolumeSpecName "kube-api-access-qchdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:17.649269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.649220    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:17.649269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.649262    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5f4282-4d1a-467e-b3c5-e0c361968222-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:17.649269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:17.649276    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qchdr\" (UniqueName: \"kubernetes.io/projected/fa5f4282-4d1a-467e-b3c5-e0c361968222-kube-api-access-qchdr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:18.309426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:18.309392    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"
Nov 04 00:07:18.309600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:18.309392    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh" event={"ID":"fa5f4282-4d1a-467e-b3c5-e0c361968222","Type":"ContainerDied","Data":"ae74facffce08a3954807ddb6f2ea134c2e40e5eb7b400b2479c8130f942cd31"}
Nov 04 00:07:18.309600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:18.309522    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae74facffce08a3954807ddb6f2ea134c2e40e5eb7b400b2479c8130f942cd31"
Nov 04 00:07:18.894053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:18.894009    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr" podUID="fccae4a8-1d52-43d1-a4c4-bd6400da574a" containerName="payload-processing" containerID="cri-o://f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86" gracePeriod=25
Nov 04 00:07:19.185442 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.185413    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:19.204149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.204115    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:07:19.272076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.272040    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials\") pod \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") "
Nov 04 00:07:19.272284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.272140    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume\") pod \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") "
Nov 04 00:07:19.272284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.272184    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume\") pod \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") "
Nov 04 00:07:19.272633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.272600    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume" pod "fccae4a8-1d52-43d1-a4c4-bd6400da574a" (UID: "fccae4a8-1d52-43d1-a4c4-bd6400da574a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:19.272760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.272609    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume" pod "fccae4a8-1d52-43d1-a4c4-bd6400da574a" (UID: "fccae4a8-1d52-43d1-a4c4-bd6400da574a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:19.272889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.272864    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt\") pod \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\" (UID: \"fccae4a8-1d52-43d1-a4c4-bd6400da574a\") "
Nov 04 00:07:19.274513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.274486    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:19.274636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.274516    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fccae4a8-1d52-43d1-a4c4-bd6400da574a-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:19.282668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.282610    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials" pod "fccae4a8-1d52-43d1-a4c4-bd6400da574a" (UID: "fccae4a8-1d52-43d1-a4c4-bd6400da574a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:19.297603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.297539    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt" pod "fccae4a8-1d52-43d1-a4c4-bd6400da574a" (UID: "fccae4a8-1d52-43d1-a4c4-bd6400da574a"). InnerVolumeSpecName "kube-api-access-4blnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:19.320137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.320096    5074 generic.go:356] "Generic (PLEG): container finished" podID="fccae4a8-1d52-43d1-a4c4-bd6400da574a" containerID="f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86" exitCode=0
Nov 04 00:07:19.320315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.320153    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"
Nov 04 00:07:19.320315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.320165    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr" event={"ID":"fccae4a8-1d52-43d1-a4c4-bd6400da574a","Type":"ContainerDied","Data":"f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86"}
Nov 04 00:07:19.320315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.320228    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr" event={"ID":"fccae4a8-1d52-43d1-a4c4-bd6400da574a","Type":"ContainerDied","Data":"f3f515c20431454b4ee93b332c0bf22c0644a7dcb8efc056fb3b477dd88e355f"}
Nov 04 00:07:19.320315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.320260    5074 scope.go:122] "RemoveContainer" containerID="f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86"
Nov 04 00:07:19.332929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.332905    5074 scope.go:122] "RemoveContainer" containerID="f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86"
Nov 04 00:07:19.333213 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:07:19.333193    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86\": container with ID starting with f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86 not found: ID does not exist" containerID="f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86"
Nov 04 00:07:19.333288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.333228    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86"} err="failed to get container status \"f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86\": rpc error: code = NotFound desc = could not find container \"f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86\": container with ID starting with f6b3a7b8754606d99be9419c70ac4bc6b9cdcc3a3a69f8736187e18f203bdf86 not found: ID does not exist"
Nov 04 00:07:19.349804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.349766    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"]
Nov 04 00:07:19.364098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.364063    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-b9f9g9kwr"]
Nov 04 00:07:19.375889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.375859    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:19.375889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:19.375885    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blnt\" (UniqueName: \"kubernetes.io/projected/fccae4a8-1d52-43d1-a4c4-bd6400da574a-kube-api-access-4blnt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:21.020858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:21.020823    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fccae4a8-1d52-43d1-a4c4-bd6400da574a" path="/var/lib/kubelet/pods/fccae4a8-1d52-43d1-a4c4-bd6400da574a/volumes"
Nov 04 00:07:25.126716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.126653    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"]
Nov 04 00:07:25.128752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.128705    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fccae4a8-1d52-43d1-a4c4-bd6400da574a" containerName="payload-processing"
Nov 04 00:07:25.128931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.128914    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccae4a8-1d52-43d1-a4c4-bd6400da574a" containerName="payload-processing"
Nov 04 00:07:25.129122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.129109    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa5f4282-4d1a-467e-b3c5-e0c361968222" containerName="revoke-keys"
Nov 04 00:07:25.129255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.129241    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5f4282-4d1a-467e-b3c5-e0c361968222" containerName="revoke-keys"
Nov 04 00:07:25.129625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.129607    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5f4282-4d1a-467e-b3c5-e0c361968222" containerName="revoke-keys"
Nov 04 00:07:25.129779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.129766    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccae4a8-1d52-43d1-a4c4-bd6400da574a" containerName="payload-processing"
Nov 04 00:07:25.136530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.136502    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.155672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.155620    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"]
Nov 04 00:07:25.161264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.161234    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"agc-cred-d7d34f0bd51b\""
Nov 04 00:07:25.161908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.161446    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"openshift-service-ca.crt\""
Nov 04 00:07:25.161908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.161533    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"kube-root-ca.crt\""
Nov 04 00:07:25.161908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.161578    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c7d218\""
Nov 04 00:07:25.164979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.164007    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"routing-overlay\""
Nov 04 00:07:25.165899 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.165882    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"payload-processing-external-model-e2e--36d07d4b-dockercfg-p2wcn\""
Nov 04 00:07:25.319970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.319612    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.319970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.319688    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.319970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.319787    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2ps\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.319970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.319859    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.421502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.421404    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.421684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.421514    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2ps\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.421684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.421592    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.421684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.421674    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.422359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.422327    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.422853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.422822    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.425702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.425676    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.437397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.437363    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2ps\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.468829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.468795    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:25.775454 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:25.775421    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619acd5c_922a_45a3_a670_896ca43bb518.slice/crio-658497f18357fb2bb277a118b9edaf652bc5b186a6901ca3fa30c561cb995b25 WatchSource:0}: Error finding container 658497f18357fb2bb277a118b9edaf652bc5b186a6901ca3fa30c561cb995b25: Status 404 returned error can't find the container with id 658497f18357fb2bb277a118b9edaf652bc5b186a6901ca3fa30c561cb995b25
Nov 04 00:07:25.779268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:25.779241    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"]
Nov 04 00:07:26.294918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:26.294884    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:26.390175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:26.390139    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc" event={"ID":"619acd5c-922a-45a3-a670-896ca43bb518","Type":"ContainerStarted","Data":"81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405"}
Nov 04 00:07:26.390339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:26.390198    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc" event={"ID":"619acd5c-922a-45a3-a670-896ca43bb518","Type":"ContainerStarted","Data":"658497f18357fb2bb277a118b9edaf652bc5b186a6901ca3fa30c561cb995b25"}
Nov 04 00:07:26.390339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:26.390243    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:26.438244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:26.438168    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc" podStartSLOduration=1.4381499500000001 podStartE2EDuration="1.43814995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:26.437699689 +0000 UTC m=+2311.537299831" watchObservedRunningTime="2026-11-04 00:07:26.43814995 +0000 UTC m=+2311.537750095"
Nov 04 00:07:26.585540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:26.585458    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"]
Nov 04 00:07:28.024997 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:07:28.024951    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:07:31.586299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.586258    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6" podUID="e6d26fce-0711-43f0-96de-97670163c914" containerName="payload-processing" containerID="cri-o://351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb" gracePeriod=25
Nov 04 00:07:31.874051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.874020    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:31.964389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.964357    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl\") pod \"e6d26fce-0711-43f0-96de-97670163c914\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") "
Nov 04 00:07:31.964585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.964506    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume\") pod \"e6d26fce-0711-43f0-96de-97670163c914\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") "
Nov 04 00:07:31.964585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.964536    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials\") pod \"e6d26fce-0711-43f0-96de-97670163c914\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") "
Nov 04 00:07:31.964585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.964571    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume\") pod \"e6d26fce-0711-43f0-96de-97670163c914\" (UID: \"e6d26fce-0711-43f0-96de-97670163c914\") "
Nov 04 00:07:31.965020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.964990    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume" pod "e6d26fce-0711-43f0-96de-97670163c914" (UID: "e6d26fce-0711-43f0-96de-97670163c914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:31.965112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.965001    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume" pod "e6d26fce-0711-43f0-96de-97670163c914" (UID: "e6d26fce-0711-43f0-96de-97670163c914"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:31.965217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.965196    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:31.965286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.965224    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d26fce-0711-43f0-96de-97670163c914-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:31.966622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.966593    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl" pod "e6d26fce-0711-43f0-96de-97670163c914" (UID: "e6d26fce-0711-43f0-96de-97670163c914"). InnerVolumeSpecName "kube-api-access-4dtcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:31.966749 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:31.966715    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials" pod "e6d26fce-0711-43f0-96de-97670163c914" (UID: "e6d26fce-0711-43f0-96de-97670163c914"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:32.066556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.066520    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtcl\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-kube-api-access-4dtcl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:32.066556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.066556    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d26fce-0711-43f0-96de-97670163c914-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:32.468663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.468623    5074 generic.go:356] "Generic (PLEG): container finished" podID="e6d26fce-0711-43f0-96de-97670163c914" containerID="351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb" exitCode=0
Nov 04 00:07:32.469014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.468696    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6" event={"ID":"e6d26fce-0711-43f0-96de-97670163c914","Type":"ContainerDied","Data":"351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb"}
Nov 04 00:07:32.469014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.468817    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6" event={"ID":"e6d26fce-0711-43f0-96de-97670163c914","Type":"ContainerDied","Data":"82c88b290f77781af918ee4579e65eb175c4ea3f592ce32e71b8b78f7c763a8d"}
Nov 04 00:07:32.469014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.468852    5074 scope.go:122] "RemoveContainer" containerID="351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb"
Nov 04 00:07:32.469435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.469409    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"
Nov 04 00:07:32.484971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.484950    5074 scope.go:122] "RemoveContainer" containerID="351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb"
Nov 04 00:07:32.485315 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:07:32.485288    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb\": container with ID starting with 351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb not found: ID does not exist" containerID="351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb"
Nov 04 00:07:32.485409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.485321    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb"} err="failed to get container status \"351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb\": rpc error: code = NotFound desc = could not find container \"351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb\": container with ID starting with 351260eef46f18502221b1163c7697e38661faebb00a3db8c941e90ae872ddfb not found: ID does not exist"
Nov 04 00:07:32.512309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.512282    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"]
Nov 04 00:07:32.540447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:32.540419    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c7dsjf6"]
Nov 04 00:07:33.022524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:33.022484    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d26fce-0711-43f0-96de-97670163c914" path="/var/lib/kubelet/pods/e6d26fce-0711-43f0-96de-97670163c914/volumes"
Nov 04 00:07:36.552095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.552059    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"]
Nov 04 00:07:36.553043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.553022    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e6d26fce-0711-43f0-96de-97670163c914" containerName="payload-processing"
Nov 04 00:07:36.553043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.553042    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d26fce-0711-43f0-96de-97670163c914" containerName="payload-processing"
Nov 04 00:07:36.553219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.553199    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d26fce-0711-43f0-96de-97670163c914" containerName="payload-processing"
Nov 04 00:07:36.558887 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.558861    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.573881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.573844    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"]
Nov 04 00:07:36.576101 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.576040    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-114abfa61a32\""
Nov 04 00:07:36.674279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.674239    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.674491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.674309    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.674491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.674444    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hp2q\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.674632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.674491    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.775931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.775897    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hp2q\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.776127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.775954    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.776210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.776161    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.776281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.776257    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.776959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.776930    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.777043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.776974    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.778706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.778686    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.792106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.792086    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hp2q\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:36.893471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:36.893389    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:37.184796 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:37.184760    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40351b31_e955_4ed6_9d83_7c12c271b7ba.slice/crio-ad51257ed9315ab214fbf7719ddd073ea74a8b47526a12c5f2701bd045d98646 WatchSource:0}: Error finding container ad51257ed9315ab214fbf7719ddd073ea74a8b47526a12c5f2701bd045d98646: Status 404 returned error can't find the container with id ad51257ed9315ab214fbf7719ddd073ea74a8b47526a12c5f2701bd045d98646
Nov 04 00:07:37.188967 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:37.188937    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"]
Nov 04 00:07:37.403849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:37.403818    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:07:37.520069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:37.520033    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk" event={"ID":"40351b31-e955-4ed6-9d83-7c12c271b7ba","Type":"ContainerStarted","Data":"2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594"}
Nov 04 00:07:37.520253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:37.520103    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:37.520253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:37.520128    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk" event={"ID":"40351b31-e955-4ed6-9d83-7c12c271b7ba","Type":"ContainerStarted","Data":"ad51257ed9315ab214fbf7719ddd073ea74a8b47526a12c5f2701bd045d98646"}
Nov 04 00:07:37.557323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:37.557261    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk" podStartSLOduration=1.5572476979999998 podStartE2EDuration="1.557247698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:37.551047621 +0000 UTC m=+2322.650647766" watchObservedRunningTime="2026-11-04 00:07:37.557247698 +0000 UTC m=+2322.656847846"
Nov 04 00:07:43.342884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.342846    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"]
Nov 04 00:07:43.351174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.351125    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.352379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.352349    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"]
Nov 04 00:07:43.364665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.364638    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"e2e-cmp-legacy-39e446-provider-cred\""
Nov 04 00:07:43.367346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.367325    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-e2e--7c418871-dockercfg-xzlvr\""
Nov 04 00:07:43.370137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.370118    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"kube-root-ca.crt\""
Nov 04 00:07:43.373021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.372996    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a002b\""
Nov 04 00:07:43.373135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.373031    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"openshift-service-ca.crt\""
Nov 04 00:07:43.375725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.375699    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"routing-overlay\""
Nov 04 00:07:43.379092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.379067    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv6t\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.379230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.379153    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.379312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.379289    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.379421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.379404    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.489167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.487694    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv6t\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.489167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.487799    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.489167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.487888    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.489167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.487957    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.489167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.489000    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.510763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.490153    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.510763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.499281    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.516874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.516802    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv6t\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.668196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.668118    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:43.986007 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:43.985978    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7532bb_68ce_4fbd_a22f_a242315d022b.slice/crio-ee6c1c9be41a19bc6aeb24cd904090bca84ac38e99883592aa2bc3cdc55e6b3a WatchSource:0}: Error finding container ee6c1c9be41a19bc6aeb24cd904090bca84ac38e99883592aa2bc3cdc55e6b3a: Status 404 returned error can't find the container with id ee6c1c9be41a19bc6aeb24cd904090bca84ac38e99883592aa2bc3cdc55e6b3a
Nov 04 00:07:43.989477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:43.989451    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"]
Nov 04 00:07:44.592599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:44.592559    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs" event={"ID":"de7532bb-68ce-4fbd-a22f-a242315d022b","Type":"ContainerStarted","Data":"6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633"}
Nov 04 00:07:44.593127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:44.592629    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs" event={"ID":"de7532bb-68ce-4fbd-a22f-a242315d022b","Type":"ContainerStarted","Data":"ee6c1c9be41a19bc6aeb24cd904090bca84ac38e99883592aa2bc3cdc55e6b3a"}
Nov 04 00:07:44.593127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:44.592658    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:44.636496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:44.636446    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs" podStartSLOduration=1.6364314389999999 podStartE2EDuration="1.636431439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:44.636161297 +0000 UTC m=+2329.735761443" watchObservedRunningTime="2026-11-04 00:07:44.636431439 +0000 UTC m=+2329.736031583"
Nov 04 00:07:48.534440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:48.534409    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:07:48.601341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:48.601307    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"]
Nov 04 00:07:53.280962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.280922    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"]
Nov 04 00:07:53.286150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.286126    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.294508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.294486    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:07:53.297596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.297572    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:07:53.302721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.302694    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"]
Nov 04 00:07:53.432404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.432362    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.432594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.432419    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8bq\" (UniqueName: \"kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.432594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.432536    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.534174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.534090    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.534174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.534139    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8bq\" (UniqueName: \"kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.534433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.534179    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.534936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.534913    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.536567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.536541    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.547654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.547629    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8bq\" (UniqueName: \"kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq\") pod \"maas-api-revoke-sub-5020b25a-2kskm\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.601661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.601609    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2" podUID="d3d27399-d72f-4401-9a4f-760831884a99" containerName="payload-processing" containerID="cri-o://d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb" gracePeriod=25
Nov 04 00:07:53.606365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.606339    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:53.886645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.886619    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:07:53.895071 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:07:53.895041    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb410d1fd_7f3a_4ff9_a6aa_5942e6ac2fbb.slice/crio-bb5228b42bb00b44592e0e7d2e82fd5a07a2611a6c667f103c3226a1048ce513 WatchSource:0}: Error finding container bb5228b42bb00b44592e0e7d2e82fd5a07a2611a6c667f103c3226a1048ce513: Status 404 returned error can't find the container with id bb5228b42bb00b44592e0e7d2e82fd5a07a2611a6c667f103c3226a1048ce513
Nov 04 00:07:53.898604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:53.898582    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"]
Nov 04 00:07:54.044021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.043935    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume\") pod \"d3d27399-d72f-4401-9a4f-760831884a99\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") "
Nov 04 00:07:54.044021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044010    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials\") pod \"d3d27399-d72f-4401-9a4f-760831884a99\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") "
Nov 04 00:07:54.044225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044198    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume\") pod \"d3d27399-d72f-4401-9a4f-760831884a99\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") "
Nov 04 00:07:54.044342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044319    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl\") pod \"d3d27399-d72f-4401-9a4f-760831884a99\" (UID: \"d3d27399-d72f-4401-9a4f-760831884a99\") "
Nov 04 00:07:54.044416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044340    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume" pod "d3d27399-d72f-4401-9a4f-760831884a99" (UID: "d3d27399-d72f-4401-9a4f-760831884a99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:54.044618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044596    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume" pod "d3d27399-d72f-4401-9a4f-760831884a99" (UID: "d3d27399-d72f-4401-9a4f-760831884a99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:54.044940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044921    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:54.045018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.044945    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d27399-d72f-4401-9a4f-760831884a99-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:54.046358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.046335    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials" pod "d3d27399-d72f-4401-9a4f-760831884a99" (UID: "d3d27399-d72f-4401-9a4f-760831884a99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:54.046446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.046424    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl" pod "d3d27399-d72f-4401-9a4f-760831884a99" (UID: "d3d27399-d72f-4401-9a4f-760831884a99"). InnerVolumeSpecName "kube-api-access-wqcnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:54.146424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.146385    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcnl\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-kube-api-access-wqcnl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:54.146424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.146414    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d27399-d72f-4401-9a4f-760831884a99-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:54.696530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.696497    5074 generic.go:356] "Generic (PLEG): container finished" podID="d3d27399-d72f-4401-9a4f-760831884a99" containerID="d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb" exitCode=0
Nov 04 00:07:54.697017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.696544    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"
Nov 04 00:07:54.697017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.696576    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2" event={"ID":"d3d27399-d72f-4401-9a4f-760831884a99","Type":"ContainerDied","Data":"d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb"}
Nov 04 00:07:54.697017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.696635    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2" event={"ID":"d3d27399-d72f-4401-9a4f-760831884a99","Type":"ContainerDied","Data":"7834e97936e0f43b4a857cdaa8460e5f96595a60653190bc2ec3f8f7032ab32e"}
Nov 04 00:07:54.697017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.696665    5074 scope.go:122] "RemoveContainer" containerID="d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb"
Nov 04 00:07:54.698659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.698619    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm" event={"ID":"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb","Type":"ContainerStarted","Data":"ecee004215112eacbc87abe11e239bc6e12610ad021f4ff88f175e77c137d152"}
Nov 04 00:07:54.698832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.698679    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm" event={"ID":"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb","Type":"ContainerStarted","Data":"bb5228b42bb00b44592e0e7d2e82fd5a07a2611a6c667f103c3226a1048ce513"}
Nov 04 00:07:54.710444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.710250    5074 scope.go:122] "RemoveContainer" containerID="d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb"
Nov 04 00:07:54.710671 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:07:54.710609    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb\": container with ID starting with d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb not found: ID does not exist" containerID="d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb"
Nov 04 00:07:54.710671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.710642    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb"} err="failed to get container status \"d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb\": rpc error: code = NotFound desc = could not find container \"d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb\": container with ID starting with d9fdc1ed457cc82bba549195ce622b2a6db7b642ec99c4d524875d48ff657ddb not found: ID does not exist"
Nov 04 00:07:54.736687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.736638    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm" podStartSLOduration=1.736625594 podStartE2EDuration="1.736625594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:07:54.7338417 +0000 UTC m=+2339.833441844" watchObservedRunningTime="2026-11-04 00:07:54.736625594 +0000 UTC m=+2339.836225755"
Nov 04 00:07:54.762673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.762632    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"]
Nov 04 00:07:54.769351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:54.769316    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96gmwv2"]
Nov 04 00:07:55.024404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:55.024319    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d27399-d72f-4401-9a4f-760831884a99" path="/var/lib/kubelet/pods/d3d27399-d72f-4401-9a4f-760831884a99/volumes"
Nov 04 00:07:55.607301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:55.607271    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:07:55.714328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:55.714290    5074 generic.go:356] "Generic (PLEG): container finished" podID="b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" containerID="ecee004215112eacbc87abe11e239bc6e12610ad021f4ff88f175e77c137d152" exitCode=0
Nov 04 00:07:55.714734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:55.714368    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm" event={"ID":"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb","Type":"ContainerDied","Data":"ecee004215112eacbc87abe11e239bc6e12610ad021f4ff88f175e77c137d152"}
Nov 04 00:07:56.900328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:56.900298    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:57.086864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.086781    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq\" (UniqueName: \"kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq\") pod \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") "
Nov 04 00:07:57.086864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.086839    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca\") pod \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") "
Nov 04 00:07:57.087113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.086982    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca\") pod \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\" (UID: \"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb\") "
Nov 04 00:07:57.087213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.087187    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca" pod "b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" (UID: "b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:57.087482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.087457    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:57.089069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.089034    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq" pod "b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" (UID: "b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb"). InnerVolumeSpecName "kube-api-access-rp8bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:57.089069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.089037    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca" pod "b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" (UID: "b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:07:57.188645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.188605    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8bq\" (UniqueName: \"kubernetes.io/projected/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-kube-api-access-rp8bq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:57.188645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.188641    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:57.732680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.732647    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"
Nov 04 00:07:57.732891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.732676    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm" event={"ID":"b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb","Type":"ContainerDied","Data":"bb5228b42bb00b44592e0e7d2e82fd5a07a2611a6c667f103c3226a1048ce513"}
Nov 04 00:07:57.732891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:07:57.732727    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb5228b42bb00b44592e0e7d2e82fd5a07a2611a6c667f103c3226a1048ce513"
Nov 04 00:07:58.023488 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:07:58.023408    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:08:03.322524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:03.322486    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"]
Nov 04 00:08:03.323184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:03.322683    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy" containerID="cri-o://4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396" gracePeriod=30
Nov 04 00:08:08.631962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.631931    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:08:08.694306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694274    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694334    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694376    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694480    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694519    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694580    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694629    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694697    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.694785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.694734    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo\") pod \"81abacba-142f-4373-b6db-62e0f6e5bd5f\" (UID: \"81abacba-142f-4373-b6db-62e0f6e5bd5f\") "
Nov 04 00:08:08.696239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.696066    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:08.696239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.696081    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:08.696444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.696259    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:08.697461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.697431    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:08.697754 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.697711    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:08.706209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.706089    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:08.706209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.706134    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:08:08.706209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.706165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "kube-api-access-8hwsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:08.706209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.706188    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token" pod "81abacba-142f-4373-b6db-62e0f6e5bd5f" (UID: "81abacba-142f-4373-b6db-62e0f6e5bd5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:08.796569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796535    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796569    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hwsr\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-kube-api-access-8hwsr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796579    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81abacba-142f-4373-b6db-62e0f6e5bd5f-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796588    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796597    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796607    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796617    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796626    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.796801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.796635    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81abacba-142f-4373-b6db-62e0f6e5bd5f-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:08.857836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.857726    5074 generic.go:356] "Generic (PLEG): container finished" podID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerID="4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396" exitCode=0
Nov 04 00:08:08.857836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.857798    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" event={"ID":"81abacba-142f-4373-b6db-62e0f6e5bd5f","Type":"ContainerDied","Data":"4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396"}
Nov 04 00:08:08.857836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.857818    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"
Nov 04 00:08:08.858086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.857870    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp" event={"ID":"81abacba-142f-4373-b6db-62e0f6e5bd5f","Type":"ContainerDied","Data":"862bc81ab06d2d447aa96bbd3d3c382d86fbc8681217f72cb7e056a729bd3368"}
Nov 04 00:08:08.858086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.857894    5074 scope.go:122] "RemoveContainer" containerID="4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396"
Nov 04 00:08:08.870185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.870161    5074 scope.go:122] "RemoveContainer" containerID="4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396"
Nov 04 00:08:08.870421 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:08:08.870403    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396\": container with ID starting with 4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396 not found: ID does not exist" containerID="4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396"
Nov 04 00:08:08.870464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.870431    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396"} err="failed to get container status \"4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396\": rpc error: code = NotFound desc = could not find container \"4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396\": container with ID starting with 4ae2982fa629dc4d8adde500815c0f35a34b1cc3a43fd4adb10924644e50e396 not found: ID does not exist"
Nov 04 00:08:08.958901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.958867    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"]
Nov 04 00:08:08.999346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:08.999311    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-rrlbp"]
Nov 04 00:08:09.023476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:09.023444    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" path="/var/lib/kubelet/pods/81abacba-142f-4373-b6db-62e0f6e5bd5f/volumes"
Nov 04 00:08:17.242779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.242723    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"]
Nov 04 00:08:17.244080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244050    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d3d27399-d72f-4401-9a4f-760831884a99" containerName="payload-processing"
Nov 04 00:08:17.244080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244080    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d27399-d72f-4401-9a4f-760831884a99" containerName="payload-processing"
Nov 04 00:08:17.244252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244103    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" containerName="revoke-keys"
Nov 04 00:08:17.244252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244111    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" containerName="revoke-keys"
Nov 04 00:08:17.244252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244137    5074 cpu_manager.go:414] "containerMap: removing container" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy"
Nov 04 00:08:17.244252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244145    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy"
Nov 04 00:08:17.244385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244338    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d27399-d72f-4401-9a4f-760831884a99" containerName="payload-processing"
Nov 04 00:08:17.244385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244356    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" containerName="revoke-keys"
Nov 04 00:08:17.244385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.244367    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="81abacba-142f-4373-b6db-62e0f6e5bd5f" containerName="istio-proxy"
Nov 04 00:08:17.250680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.250634    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.258044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.258018    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"]
Nov 04 00:08:17.264414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.264388    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-93c9f2d551bc\""
Nov 04 00:08:17.331258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.331214    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.331421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.331353    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.331421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.331396    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.331421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.331414    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd69h\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.432879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.432835    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.433079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.432907    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.433079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.432952    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd69h\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.433255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.433237    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.433779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.433734    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.434156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.434131    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.435835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.435809    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.678793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.678697    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd69h\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:17.869538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:17.869505    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:18.164135 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:08:18.164092    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc652bf3c_a1da_4201_8b00_09166bb4ed73.slice/crio-0bf8c92fbbef998f17a0bdfe9bd20325f8c931036d124312479b3e5da2215aaf WatchSource:0}: Error finding container 0bf8c92fbbef998f17a0bdfe9bd20325f8c931036d124312479b3e5da2215aaf: Status 404 returned error can't find the container with id 0bf8c92fbbef998f17a0bdfe9bd20325f8c931036d124312479b3e5da2215aaf
Nov 04 00:08:18.170505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:18.170480    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"]
Nov 04 00:08:19.021678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:19.021644    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:19.021678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:19.021680    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn" event={"ID":"c652bf3c-a1da-4201-8b00-09166bb4ed73","Type":"ContainerStarted","Data":"729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2"}
Nov 04 00:08:19.022254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:19.021698    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn" event={"ID":"c652bf3c-a1da-4201-8b00-09166bb4ed73","Type":"ContainerStarted","Data":"0bf8c92fbbef998f17a0bdfe9bd20325f8c931036d124312479b3e5da2215aaf"}
Nov 04 00:08:22.822301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.822247    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn" podStartSLOduration=5.822232148 podStartE2EDuration="5.822232148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:08:19.089036148 +0000 UTC m=+2364.188636293" watchObservedRunningTime="2026-11-04 00:08:22.822232148 +0000 UTC m=+2367.921832291"
Nov 04 00:08:22.823418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.823393    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"]
Nov 04 00:08:22.828213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.828194    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:22.840595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.840569    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"]
Nov 04 00:08:22.843489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.843463    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"agc-cred-836dc2249c00\""
Nov 04 00:08:22.843607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.843466    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"payload-processing-external-model-e2e--58feeaa8-dockercfg-sqpx4\""
Nov 04 00:08:22.846080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.846060    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"routing-overlay\""
Nov 04 00:08:22.848780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.848714    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"payload-processing-external-model-plugins-e2e-extmod-a-86deca\""
Nov 04 00:08:22.851423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.851402    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"openshift-service-ca.crt\""
Nov 04 00:08:22.851644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.851606    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"kube-root-ca.crt\""
Nov 04 00:08:22.916432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.916398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:22.916926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.916725    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:22.919594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.919569    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:22.919840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:22.919822    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klphh\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.036762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.032704    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.036762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.032806    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klphh\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.036762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.032961    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.036762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.033041    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.036762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.034187    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.045772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.043589    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.065384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.064418    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.067186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.067110    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klphh\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.148984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.148892    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:23.603184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.599794    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"]
Nov 04 00:08:23.618881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.615895    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"]
Nov 04 00:08:23.618881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.615941    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"]
Nov 04 00:08:23.618881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.616088    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.643508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.643219    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-86deca\"/\"agc-cred-589a1d4fcae0\""
Nov 04 00:08:23.757082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.757046    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.757235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.757100    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5w9\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.757330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.757302    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.757476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.757458    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.858586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.858504    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.859052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.858643    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.859052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.858806    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.859052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.858853    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5w9\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.859488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.859456    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.859488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.859465    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.861512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.861491    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.874863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.874841    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5w9\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:23.956302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:23.956274    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:24.094875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:24.090043    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m" event={"ID":"a750275d-0691-4f27-8973-53679397b28c","Type":"ContainerStarted","Data":"8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e"}
Nov 04 00:08:24.094875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:24.090107    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m" event={"ID":"a750275d-0691-4f27-8973-53679397b28c","Type":"ContainerStarted","Data":"c8e54051353ada6214d11d3335827285d1fdfb0ed9596f9856bc9b9036e35da2"}
Nov 04 00:08:24.094875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:24.090154    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:24.121069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:24.120954    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m" podStartSLOduration=2.1209357620000002 podStartE2EDuration="2.120935762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:08:24.119310501 +0000 UTC m=+2369.218910646" watchObservedRunningTime="2026-11-04 00:08:24.120935762 +0000 UTC m=+2369.220535904"
Nov 04 00:08:24.272303 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:08:24.272261    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5256dac2_e5d9_415d_b4bc_6b5860844a2a.slice/crio-e5aa690de9fc8935b715a91fe9906d6915dbec46cc3290a3bdec21847ec946fa WatchSource:0}: Error finding container e5aa690de9fc8935b715a91fe9906d6915dbec46cc3290a3bdec21847ec946fa: Status 404 returned error can't find the container with id e5aa690de9fc8935b715a91fe9906d6915dbec46cc3290a3bdec21847ec946fa
Nov 04 00:08:24.275050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:24.274995    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"]
Nov 04 00:08:25.099952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:25.099913    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9" event={"ID":"5256dac2-e5d9-415d-b4bc-6b5860844a2a","Type":"ContainerStarted","Data":"89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1"}
Nov 04 00:08:25.100462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:25.099983    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9" event={"ID":"5256dac2-e5d9-415d-b4bc-6b5860844a2a","Type":"ContainerStarted","Data":"e5aa690de9fc8935b715a91fe9906d6915dbec46cc3290a3bdec21847ec946fa"}
Nov 04 00:08:25.100462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:25.100016    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:25.133168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:25.133112    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9" podStartSLOduration=2.1330980410000002 podStartE2EDuration="2.133098041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:08:25.127028756 +0000 UTC m=+2370.226628900" watchObservedRunningTime="2026-11-04 00:08:25.133098041 +0000 UTC m=+2370.232698234"
Nov 04 00:08:28.068791 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:08:28.068731    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:08:30.032004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:30.031970    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:08:30.109717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:30.109680    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"]
Nov 04 00:08:35.104655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.104625    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:35.111000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.110972    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk" podUID="40351b31-e955-4ed6-9d83-7c12c271b7ba" containerName="payload-processing" containerID="cri-o://2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594" gracePeriod=25
Nov 04 00:08:35.412068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.412038    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:08:35.421677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.421650    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials\") pod \"40351b31-e955-4ed6-9d83-7c12c271b7ba\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") "
Nov 04 00:08:35.421864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.421845    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q\") pod \"40351b31-e955-4ed6-9d83-7c12c271b7ba\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") "
Nov 04 00:08:35.421924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.421912    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume\") pod \"40351b31-e955-4ed6-9d83-7c12c271b7ba\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") "
Nov 04 00:08:35.421962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.421941    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume\") pod \"40351b31-e955-4ed6-9d83-7c12c271b7ba\" (UID: \"40351b31-e955-4ed6-9d83-7c12c271b7ba\") "
Nov 04 00:08:35.422352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.422321    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume" pod "40351b31-e955-4ed6-9d83-7c12c271b7ba" (UID: "40351b31-e955-4ed6-9d83-7c12c271b7ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:35.422547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.422396    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume" pod "40351b31-e955-4ed6-9d83-7c12c271b7ba" (UID: "40351b31-e955-4ed6-9d83-7c12c271b7ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:35.422620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.422603    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:35.422668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.422623    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40351b31-e955-4ed6-9d83-7c12c271b7ba-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:35.424491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.424463    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q" pod "40351b31-e955-4ed6-9d83-7c12c271b7ba" (UID: "40351b31-e955-4ed6-9d83-7c12c271b7ba"). InnerVolumeSpecName "kube-api-access-9hp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:35.424491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.424476    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials" pod "40351b31-e955-4ed6-9d83-7c12c271b7ba" (UID: "40351b31-e955-4ed6-9d83-7c12c271b7ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:35.524124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.524084    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hp2q\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-kube-api-access-9hp2q\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:35.524124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:35.524117    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40351b31-e955-4ed6-9d83-7c12c271b7ba-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:36.112235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.112201    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:08:36.197433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.196810    5074 generic.go:356] "Generic (PLEG): container finished" podID="40351b31-e955-4ed6-9d83-7c12c271b7ba" containerID="2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594" exitCode=0
Nov 04 00:08:36.197433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.196887    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"
Nov 04 00:08:36.197433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.196934    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk" event={"ID":"40351b31-e955-4ed6-9d83-7c12c271b7ba","Type":"ContainerDied","Data":"2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594"}
Nov 04 00:08:36.197433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.197107    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk" event={"ID":"40351b31-e955-4ed6-9d83-7c12c271b7ba","Type":"ContainerDied","Data":"ad51257ed9315ab214fbf7719ddd073ea74a8b47526a12c5f2701bd045d98646"}
Nov 04 00:08:36.197433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.197155    5074 scope.go:122] "RemoveContainer" containerID="2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594"
Nov 04 00:08:36.204048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.203830    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"]
Nov 04 00:08:36.216465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.216213    5074 scope.go:122] "RemoveContainer" containerID="2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594"
Nov 04 00:08:36.216716 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:08:36.216683    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594\": container with ID starting with 2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594 not found: ID does not exist" containerID="2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594"
Nov 04 00:08:36.216846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.216715    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594"} err="failed to get container status \"2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594\": rpc error: code = NotFound desc = could not find container \"2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594\": container with ID starting with 2661bf2bf2bd42f0421ae0000d71ad7e54aa064fa137bb4686bf3776c17bb594 not found: ID does not exist"
Nov 04 00:08:36.265567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.265536    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"]
Nov 04 00:08:36.302383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:36.302340    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c749bmrjk"]
Nov 04 00:08:37.020990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:37.020957    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40351b31-e955-4ed6-9d83-7c12c271b7ba" path="/var/lib/kubelet/pods/40351b31-e955-4ed6-9d83-7c12c271b7ba/volumes"
Nov 04 00:08:41.204272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.204212    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m" podUID="a750275d-0691-4f27-8973-53679397b28c" containerName="payload-processing" containerID="cri-o://8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e" gracePeriod=25
Nov 04 00:08:41.510607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.510579    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:41.611544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.611512    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume\") pod \"a750275d-0691-4f27-8973-53679397b28c\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") "
Nov 04 00:08:41.611753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.611562    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials\") pod \"a750275d-0691-4f27-8973-53679397b28c\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") "
Nov 04 00:08:41.611753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.611712    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh\") pod \"a750275d-0691-4f27-8973-53679397b28c\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") "
Nov 04 00:08:41.611891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.611809    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume\") pod \"a750275d-0691-4f27-8973-53679397b28c\" (UID: \"a750275d-0691-4f27-8973-53679397b28c\") "
Nov 04 00:08:41.612028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.611999    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume" pod "a750275d-0691-4f27-8973-53679397b28c" (UID: "a750275d-0691-4f27-8973-53679397b28c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:41.612470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.612446    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume" pod "a750275d-0691-4f27-8973-53679397b28c" (UID: "a750275d-0691-4f27-8973-53679397b28c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:41.612594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.612517    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:41.614188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.614163    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials" pod "a750275d-0691-4f27-8973-53679397b28c" (UID: "a750275d-0691-4f27-8973-53679397b28c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:41.614294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.614200    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh" pod "a750275d-0691-4f27-8973-53679397b28c" (UID: "a750275d-0691-4f27-8973-53679397b28c"). InnerVolumeSpecName "kube-api-access-klphh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:41.714707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.714665    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klphh\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-kube-api-access-klphh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:41.714707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.714707    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a750275d-0691-4f27-8973-53679397b28c-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:41.714955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:41.714723    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a750275d-0691-4f27-8973-53679397b28c-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:42.254963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.254924    5074 generic.go:356] "Generic (PLEG): container finished" podID="a750275d-0691-4f27-8973-53679397b28c" containerID="8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e" exitCode=0
Nov 04 00:08:42.255390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.254973    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"
Nov 04 00:08:42.255390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.255011    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m" event={"ID":"a750275d-0691-4f27-8973-53679397b28c","Type":"ContainerDied","Data":"8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e"}
Nov 04 00:08:42.255390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.255067    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m" event={"ID":"a750275d-0691-4f27-8973-53679397b28c","Type":"ContainerDied","Data":"c8e54051353ada6214d11d3335827285d1fdfb0ed9596f9856bc9b9036e35da2"}
Nov 04 00:08:42.255390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.255092    5074 scope.go:122] "RemoveContainer" containerID="8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e"
Nov 04 00:08:42.266684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.266667    5074 scope.go:122] "RemoveContainer" containerID="8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e"
Nov 04 00:08:42.266959 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:08:42.266941    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e\": container with ID starting with 8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e not found: ID does not exist" containerID="8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e"
Nov 04 00:08:42.267038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.266965    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e"} err="failed to get container status \"8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e\": rpc error: code = NotFound desc = could not find container \"8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e\": container with ID starting with 8cbf0f0b4de5176ea93a23f74a15ae13908ecc84fd18d2c574d01c4792db856e not found: ID does not exist"
Nov 04 00:08:42.291998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.291972    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"]
Nov 04 00:08:42.305695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:42.305670    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf76kv4m"]
Nov 04 00:08:43.022933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:43.022899    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a750275d-0691-4f27-8973-53679397b28c" path="/var/lib/kubelet/pods/a750275d-0691-4f27-8973-53679397b28c/volumes"
Nov 04 00:08:55.666644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:55.666595    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:08:55.681394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:55.681368    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:08:58.022723 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:08:58.022684    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:08:59.245459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.245424    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"]
Nov 04 00:08:59.246757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.246719    5074 cpu_manager.go:414] "containerMap: removing container" podUID="40351b31-e955-4ed6-9d83-7c12c271b7ba" containerName="payload-processing"
Nov 04 00:08:59.246832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.246757    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40351b31-e955-4ed6-9d83-7c12c271b7ba" containerName="payload-processing"
Nov 04 00:08:59.246832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.246799    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a750275d-0691-4f27-8973-53679397b28c" containerName="payload-processing"
Nov 04 00:08:59.246832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.246807    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a750275d-0691-4f27-8973-53679397b28c" containerName="payload-processing"
Nov 04 00:08:59.246997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.246984    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a750275d-0691-4f27-8973-53679397b28c" containerName="payload-processing"
Nov 04 00:08:59.247033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.247014    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="40351b31-e955-4ed6-9d83-7c12c271b7ba" containerName="payload-processing"
Nov 04 00:08:59.256409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.256382    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"]
Nov 04 00:08:59.256556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.256507    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.268167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.267411    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:08:59.269829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.269805    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:08:59.378428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.378391    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.378609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.378521    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.378720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.378698    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56hm\" (UniqueName: \"kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.479808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.479770    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.480020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.479922    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56hm\" (UniqueName: \"kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.480097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.480034    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.480693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.480668    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.482259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.482231    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.493645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.493623    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56hm\" (UniqueName: \"kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm\") pod \"maas-api-revoke-sub-db80dd92-8zrhn\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.575592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.575562    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:08:59.879981 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:08:59.879949    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef7b6cc_8bae_498b_b803_70bb9d931ca6.slice/crio-d549f2e9fd8fc66bc3f1c3f6e4a14b6d8ac415195a0b35a99c97a13e963cf407 WatchSource:0}: Error finding container d549f2e9fd8fc66bc3f1c3f6e4a14b6d8ac415195a0b35a99c97a13e963cf407: Status 404 returned error can't find the container with id d549f2e9fd8fc66bc3f1c3f6e4a14b6d8ac415195a0b35a99c97a13e963cf407
Nov 04 00:08:59.882975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:08:59.882948    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"]
Nov 04 00:09:00.408385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:00.408348    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn" event={"ID":"eef7b6cc-8bae-498b-b803-70bb9d931ca6","Type":"ContainerStarted","Data":"d589a17d5c732f731ac60592b43bbfae7ba8e47a844f3dbc1a406fb56a92f914"}
Nov 04 00:09:00.408839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:00.408417    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn" event={"ID":"eef7b6cc-8bae-498b-b803-70bb9d931ca6","Type":"ContainerStarted","Data":"d549f2e9fd8fc66bc3f1c3f6e4a14b6d8ac415195a0b35a99c97a13e963cf407"}
Nov 04 00:09:00.444036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:00.443945    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn" podStartSLOduration=1.443927844 podStartE2EDuration="1.443927844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:00.442294105 +0000 UTC m=+2405.541894250" watchObservedRunningTime="2026-11-04 00:09:00.443927844 +0000 UTC m=+2405.543527990"
Nov 04 00:09:01.418277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:01.418238    5074 generic.go:356] "Generic (PLEG): container finished" podID="eef7b6cc-8bae-498b-b803-70bb9d931ca6" containerID="d589a17d5c732f731ac60592b43bbfae7ba8e47a844f3dbc1a406fb56a92f914" exitCode=0
Nov 04 00:09:01.418711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:01.418339    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn" event={"ID":"eef7b6cc-8bae-498b-b803-70bb9d931ca6","Type":"ContainerDied","Data":"d589a17d5c732f731ac60592b43bbfae7ba8e47a844f3dbc1a406fb56a92f914"}
Nov 04 00:09:02.607513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.607484    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:09:02.730555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.730512    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm\" (UniqueName: \"kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm\") pod \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") "
Nov 04 00:09:02.730784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.730658    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca\") pod \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") "
Nov 04 00:09:02.730784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.730710    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca\") pod \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\" (UID: \"eef7b6cc-8bae-498b-b803-70bb9d931ca6\") "
Nov 04 00:09:02.731190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.731157    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca" pod "eef7b6cc-8bae-498b-b803-70bb9d931ca6" (UID: "eef7b6cc-8bae-498b-b803-70bb9d931ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:02.731470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.731447    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:02.733281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.733247    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm" pod "eef7b6cc-8bae-498b-b803-70bb9d931ca6" (UID: "eef7b6cc-8bae-498b-b803-70bb9d931ca6"). InnerVolumeSpecName "kube-api-access-f56hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:02.733404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.733310    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca" pod "eef7b6cc-8bae-498b-b803-70bb9d931ca6" (UID: "eef7b6cc-8bae-498b-b803-70bb9d931ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:09:02.832712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.832671    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f56hm\" (UniqueName: \"kubernetes.io/projected/eef7b6cc-8bae-498b-b803-70bb9d931ca6-kube-api-access-f56hm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:02.832712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:02.832713    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7b6cc-8bae-498b-b803-70bb9d931ca6-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:03.439888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:03.439858    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"
Nov 04 00:09:03.440063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:03.439893    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn" event={"ID":"eef7b6cc-8bae-498b-b803-70bb9d931ca6","Type":"ContainerDied","Data":"d549f2e9fd8fc66bc3f1c3f6e4a14b6d8ac415195a0b35a99c97a13e963cf407"}
Nov 04 00:09:03.440063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:03.439955    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d549f2e9fd8fc66bc3f1c3f6e4a14b6d8ac415195a0b35a99c97a13e963cf407"
Nov 04 00:09:07.292547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:07.292512    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"]
Nov 04 00:09:07.293077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:07.292865    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy" containerID="cri-o://2df3083dfa6fe33e5e73100dd50035c13c6da7b3a3f07612df1702603ac1022a" gracePeriod=30
Nov 04 00:09:11.169448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:11.169414    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"]
Nov 04 00:09:12.568497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.568438    5074 generic.go:356] "Generic (PLEG): container finished" podID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerID="2df3083dfa6fe33e5e73100dd50035c13c6da7b3a3f07612df1702603ac1022a" exitCode=0
Nov 04 00:09:12.569112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.568507    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" event={"ID":"1f3eb966-8254-46ab-ac6d-e609f11b651c","Type":"ContainerDied","Data":"2df3083dfa6fe33e5e73100dd50035c13c6da7b3a3f07612df1702603ac1022a"}
Nov 04 00:09:12.569112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.568563    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2" event={"ID":"1f3eb966-8254-46ab-ac6d-e609f11b651c","Type":"ContainerDied","Data":"6cafdb449ec3aa63c7834c1488948b2f6044dd6472c4d30f5102a79fde73131f"}
Nov 04 00:09:12.569112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.568583    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cafdb449ec3aa63c7834c1488948b2f6044dd6472c4d30f5102a79fde73131f"
Nov 04 00:09:12.590872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.590842    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 04 00:09:12.682523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.682720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682603    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.682720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682697    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.682877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682814    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.682877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682853    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.682991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682905    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.682991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.682981    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.683106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.683053    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.683106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.683095    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket\") pod \"1f3eb966-8254-46ab-ac6d-e609f11b651c\" (UID: \"1f3eb966-8254-46ab-ac6d-e609f11b651c\") "
Nov 04 00:09:12.683751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.683710    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:12.683966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.683921    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:12.684110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.684092    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:12.684268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.684247    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:12.684496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.684479    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:12.685534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.685505    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:12.685629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.685603    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "kube-api-access-fpnsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:12.686127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.686100    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:09:12.686231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.686188    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy" pod "1f3eb966-8254-46ab-ac6d-e609f11b651c" (UID: "1f3eb966-8254-46ab-ac6d-e609f11b651c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:12.785378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785343    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785377    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785387    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3eb966-8254-46ab-ac6d-e609f11b651c-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785395    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785404    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785413    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785421    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785429    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3eb966-8254-46ab-ac6d-e609f11b651c-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:12.785628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:12.785438    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnsc\" (UniqueName: \"kubernetes.io/projected/1f3eb966-8254-46ab-ac6d-e609f11b651c-kube-api-access-fpnsc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:13.578891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.578854    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"
Nov 04 00:09:13.616882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.616837    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"]
Nov 04 00:09:13.621109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.621080    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-z76f2"]
Nov 04 00:09:13.764485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.764452    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"]
Nov 04 00:09:13.765294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.765275    5074 cpu_manager.go:414] "containerMap: removing container" podUID="eef7b6cc-8bae-498b-b803-70bb9d931ca6" containerName="revoke-keys"
Nov 04 00:09:13.765355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.765295    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef7b6cc-8bae-498b-b803-70bb9d931ca6" containerName="revoke-keys"
Nov 04 00:09:13.765355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.765319    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy"
Nov 04 00:09:13.765355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.765325    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy"
Nov 04 00:09:13.765462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.765427    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" containerName="istio-proxy"
Nov 04 00:09:13.765462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.765443    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef7b6cc-8bae-498b-b803-70bb9d931ca6" containerName="revoke-keys"
Nov 04 00:09:13.770480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.770454    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:13.778120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.778097    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"]
Nov 04 00:09:13.806156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.806135    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"agc-cred-bc347779168e\""
Nov 04 00:09:13.902027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.901950    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt689\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:13.902164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.902140    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:13.902248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.902231    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:13.902296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:13.902282    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.003900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.003869    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.004102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.004023    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt689\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.004158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.004108    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.004220 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.004156    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.004833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.004796    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.005022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.004995    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.006975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.006946    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.017603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.017575    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt689\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.091783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.091730    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:14.275759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.264836    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"]
Nov 04 00:09:14.275759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.274495    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"]
Nov 04 00:09:14.275759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.274613    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.369788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.369010    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"agc-cred-66f019330846\""
Nov 04 00:09:14.374970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.372921    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"payload-processing-external-model-e2e--4ce224e7-dockercfg-dcr7b\""
Nov 04 00:09:14.447076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.447039    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.447275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.447158    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrnfh\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.447275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.447227    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.447396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.447312    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.548832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.548696    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.548832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.548806    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrnfh\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.549084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.548861    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.549084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.548952    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.549615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.549580    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.549809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.549706    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.551646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.551623    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.563015 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.562990    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrnfh\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.610913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.610880    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"]
Nov 04 00:09:14.626718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.626693    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:14.913635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.913557    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"]
Nov 04 00:09:14.913910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.913822    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" podUID="89f9b005-a880-4895-8b97-b9d59aec49b4" containerName="discovery" containerID="cri-o://d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059" gracePeriod=30
Nov 04 00:09:14.946666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:14.946629    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"]
Nov 04 00:09:15.036781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.036717    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3eb966-8254-46ab-ac6d-e609f11b651c" path="/var/lib/kubelet/pods/1f3eb966-8254-46ab-ac6d-e609f11b651c/volumes"
Nov 04 00:09:15.230593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.230559    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 04 00:09:15.359948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.359909    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.360166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.359964    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.360166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.360024    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.360166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.360059    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.360166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.360092    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.360453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.360318    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.360546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.360513    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:15.360624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.360567    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert\") pod \"89f9b005-a880-4895-8b97-b9d59aec49b4\" (UID: \"89f9b005-a880-4895-8b97-b9d59aec49b4\") "
Nov 04 00:09:15.361405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.361375    5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-ca-configmap\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.364124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.364087    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:09:15.364255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.364099    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:15.364433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.364408    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "kube-api-access-j86wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:15.364521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.364475    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:09:15.365003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.364982    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:15.366786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.366754    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert" pod "89f9b005-a880-4895-8b97-b9d59aec49b4" (UID: "89f9b005-a880-4895-8b97-b9d59aec49b4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:09:15.463331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.463248    5074 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f9b005-a880-4895-8b97-b9d59aec49b4-local-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.463331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.463280    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.463331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.463291    5074 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-kubeconfig\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.463331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.463306    5074 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-cacerts\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.463331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.463318    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86wl\" (UniqueName: \"kubernetes.io/projected/89f9b005-a880-4895-8b97-b9d59aec49b4-kube-api-access-j86wl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.463331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.463328    5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89f9b005-a880-4895-8b97-b9d59aec49b4-istio-csr-dns-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:15.596084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.596042    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d" event={"ID":"94568fc7-11ca-4918-b7a7-37b89ef96227","Type":"ContainerStarted","Data":"72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c"}
Nov 04 00:09:15.596270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.596138    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d" event={"ID":"94568fc7-11ca-4918-b7a7-37b89ef96227","Type":"ContainerStarted","Data":"c2b5bfbec709e03f5db9fafd69db728beaf50d3b68344aa11c91408148023b81"}
Nov 04 00:09:15.596270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.596167    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:15.597517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.597484    5074 generic.go:356] "Generic (PLEG): container finished" podID="89f9b005-a880-4895-8b97-b9d59aec49b4" containerID="d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059" exitCode=0
Nov 04 00:09:15.597654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.597542    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"
Nov 04 00:09:15.597654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.597561    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" event={"ID":"89f9b005-a880-4895-8b97-b9d59aec49b4","Type":"ContainerDied","Data":"d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059"}
Nov 04 00:09:15.597654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.597610    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs" event={"ID":"89f9b005-a880-4895-8b97-b9d59aec49b4","Type":"ContainerDied","Data":"5d62f413df7823c024fab7814a5287c8459332ab6428545456545a48f8ec089a"}
Nov 04 00:09:15.597654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.597633    5074 scope.go:122] "RemoveContainer" containerID="d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059"
Nov 04 00:09:15.599316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.599291    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk" event={"ID":"33ee3776-17e7-4bbd-9c4d-96c7d633bceb","Type":"ContainerStarted","Data":"c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d"}
Nov 04 00:09:15.599472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.599328    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk" event={"ID":"33ee3776-17e7-4bbd-9c4d-96c7d633bceb","Type":"ContainerStarted","Data":"6f0d8379bc780176058b7370ea337eaa16455bd3a9b84f52a269941cce3fde80"}
Nov 04 00:09:15.599930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.599911    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:15.615590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.615490    5074 scope.go:122] "RemoveContainer" containerID="d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059"
Nov 04 00:09:15.615982 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:15.615855    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059\": container with ID starting with d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059 not found: ID does not exist" containerID="d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059"
Nov 04 00:09:15.615982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.615880    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059"} err="failed to get container status \"d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059\": rpc error: code = NotFound desc = could not find container \"d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059\": container with ID starting with d50edf2ff0eae7ed8c1f232b736f3933dabd6f43661396c6157e06d2f67ca059 not found: ID does not exist"
Nov 04 00:09:15.631492 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.631447    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d" podStartSLOduration=2.631431871 podStartE2EDuration="2.631431871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:15.623542437 +0000 UTC m=+2420.723142581" watchObservedRunningTime="2026-11-04 00:09:15.631431871 +0000 UTC m=+2420.731032015"
Nov 04 00:09:15.660608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.660551    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk" podStartSLOduration=1.660534435 podStartE2EDuration="1.660534435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:15.655565212 +0000 UTC m=+2420.755165355" watchObservedRunningTime="2026-11-04 00:09:15.660534435 +0000 UTC m=+2420.760134580"
Nov 04 00:09:15.692130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.692101    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"]
Nov 04 00:09:15.708863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:15.708833    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-497vs"]
Nov 04 00:09:16.170455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.170414    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5" podUID="92fab3cd-b25a-44bc-b85d-30efb373a0b6" containerName="payload-processing" containerID="cri-o://4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5" gracePeriod=25
Nov 04 00:09:16.468298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.468265    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:09:16.578694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.578661    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6\") pod \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") "
Nov 04 00:09:16.578926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.578735    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume\") pod \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") "
Nov 04 00:09:16.578926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.578786    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume\") pod \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") "
Nov 04 00:09:16.579128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.579104    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials\") pod \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\" (UID: \"92fab3cd-b25a-44bc-b85d-30efb373a0b6\") "
Nov 04 00:09:16.579208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.579104    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume" pod "92fab3cd-b25a-44bc-b85d-30efb373a0b6" (UID: "92fab3cd-b25a-44bc-b85d-30efb373a0b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:16.579262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.579223    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume" pod "92fab3cd-b25a-44bc-b85d-30efb373a0b6" (UID: "92fab3cd-b25a-44bc-b85d-30efb373a0b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:16.579667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.579634    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:16.579667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.579656    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fab3cd-b25a-44bc-b85d-30efb373a0b6-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:16.581316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.581278    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6" pod "92fab3cd-b25a-44bc-b85d-30efb373a0b6" (UID: "92fab3cd-b25a-44bc-b85d-30efb373a0b6"). InnerVolumeSpecName "kube-api-access-k7cx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:16.581430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.581321    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials" pod "92fab3cd-b25a-44bc-b85d-30efb373a0b6" (UID: "92fab3cd-b25a-44bc-b85d-30efb373a0b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:16.614049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.614015    5074 generic.go:356] "Generic (PLEG): container finished" podID="92fab3cd-b25a-44bc-b85d-30efb373a0b6" containerID="4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5" exitCode=0
Nov 04 00:09:16.614283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.614060    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"
Nov 04 00:09:16.614283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.614098    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5" event={"ID":"92fab3cd-b25a-44bc-b85d-30efb373a0b6","Type":"ContainerDied","Data":"4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5"}
Nov 04 00:09:16.614283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.614170    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5" event={"ID":"92fab3cd-b25a-44bc-b85d-30efb373a0b6","Type":"ContainerDied","Data":"84cc8b0c314078193c10510ea864c41bd2841ce5ae44ff234cd21833cc4590bd"}
Nov 04 00:09:16.614283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.614203    5074 scope.go:122] "RemoveContainer" containerID="4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5"
Nov 04 00:09:16.627398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.627382    5074 scope.go:122] "RemoveContainer" containerID="4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5"
Nov 04 00:09:16.627711 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:16.627681    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5\": container with ID starting with 4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5 not found: ID does not exist" containerID="4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5"
Nov 04 00:09:16.627786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.627705    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5"} err="failed to get container status \"4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5\": rpc error: code = NotFound desc = could not find container \"4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5\": container with ID starting with 4e294ee31e1f90eed48c5fc3c12df300c89e400606677aa0650b013554caefa5 not found: ID does not exist"
Nov 04 00:09:16.678045 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.678013    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"]
Nov 04 00:09:16.681076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.681054    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:16.681155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.681081    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cx6\" (UniqueName: \"kubernetes.io/projected/92fab3cd-b25a-44bc-b85d-30efb373a0b6-kube-api-access-k7cx6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:16.692338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:16.692313    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-85bf7kgj5"]
Nov 04 00:09:17.020242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.020168    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f9b005-a880-4895-8b97-b9d59aec49b4" path="/var/lib/kubelet/pods/89f9b005-a880-4895-8b97-b9d59aec49b4/volumes"
Nov 04 00:09:17.020609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.020595    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fab3cd-b25a-44bc-b85d-30efb373a0b6" path="/var/lib/kubelet/pods/92fab3cd-b25a-44bc-b85d-30efb373a0b6/volumes"
Nov 04 00:09:17.466830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.466798    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"]
Nov 04 00:09:17.467712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.467691    5074 cpu_manager.go:414] "containerMap: removing container" podUID="89f9b005-a880-4895-8b97-b9d59aec49b4" containerName="discovery"
Nov 04 00:09:17.467712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.467709    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f9b005-a880-4895-8b97-b9d59aec49b4" containerName="discovery"
Nov 04 00:09:17.467936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.467756    5074 cpu_manager.go:414] "containerMap: removing container" podUID="92fab3cd-b25a-44bc-b85d-30efb373a0b6" containerName="payload-processing"
Nov 04 00:09:17.467936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.467764    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fab3cd-b25a-44bc-b85d-30efb373a0b6" containerName="payload-processing"
Nov 04 00:09:17.468062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.467961    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fab3cd-b25a-44bc-b85d-30efb373a0b6" containerName="payload-processing"
Nov 04 00:09:17.468062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.467992    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f9b005-a880-4895-8b97-b9d59aec49b4" containerName="discovery"
Nov 04 00:09:17.473774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.473723    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"]
Nov 04 00:09:17.473915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.473828    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.484465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.484074    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:09:17.486676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.486655    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:09:17.592918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.592876    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.593102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.593044    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.593197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.593104    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6t7\" (UniqueName: \"kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.694552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.694512    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.694990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.694568    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6t7\" (UniqueName: \"kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.694990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.694776    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.695388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.695361    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.696943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.696913    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.729148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.729082    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6t7\" (UniqueName: \"kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7\") pod \"maas-api-revoke-sub-5daa7d31-mvbvz\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:17.792037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:17.792011    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:18.083687 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:09:18.083654    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode803824d_f79b_40d7_8e8e_1f7da989149f.slice/crio-af4b9fe5d51dd0e91981ad62e1d8358bfb846ced8e613baff61daf69fdbf2846 WatchSource:0}: Error finding container af4b9fe5d51dd0e91981ad62e1d8358bfb846ced8e613baff61daf69fdbf2846: Status 404 returned error can't find the container with id af4b9fe5d51dd0e91981ad62e1d8358bfb846ced8e613baff61daf69fdbf2846
Nov 04 00:09:18.086446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:18.086421    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"]
Nov 04 00:09:18.641638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:18.641595    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz" event={"ID":"e803824d-f79b-40d7-8e8e-1f7da989149f","Type":"ContainerStarted","Data":"301b95151b543f6424a52ec99d70f1975d1f5c72f9c95ba280aefac929685a2e"}
Nov 04 00:09:18.641832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:18.641695    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz" event={"ID":"e803824d-f79b-40d7-8e8e-1f7da989149f","Type":"ContainerStarted","Data":"af4b9fe5d51dd0e91981ad62e1d8358bfb846ced8e613baff61daf69fdbf2846"}
Nov 04 00:09:18.910497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:18.910439    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz" podStartSLOduration=1.910422713 podStartE2EDuration="1.910422713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:18.669991292 +0000 UTC m=+2423.769591435" watchObservedRunningTime="2026-11-04 00:09:18.910422713 +0000 UTC m=+2424.010022879"
Nov 04 00:09:19.651065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:19.650974    5074 generic.go:356] "Generic (PLEG): container finished" podID="e803824d-f79b-40d7-8e8e-1f7da989149f" containerID="301b95151b543f6424a52ec99d70f1975d1f5c72f9c95ba280aefac929685a2e" exitCode=0
Nov 04 00:09:19.651065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:19.651017    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz" event={"ID":"e803824d-f79b-40d7-8e8e-1f7da989149f","Type":"ContainerDied","Data":"301b95151b543f6424a52ec99d70f1975d1f5c72f9c95ba280aefac929685a2e"}
Nov 04 00:09:20.851785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.851762    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:20.943978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.943945    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7\" (UniqueName: \"kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7\") pod \"e803824d-f79b-40d7-8e8e-1f7da989149f\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") "
Nov 04 00:09:20.944170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.944048    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca\") pod \"e803824d-f79b-40d7-8e8e-1f7da989149f\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") "
Nov 04 00:09:20.944170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.944097    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca\") pod \"e803824d-f79b-40d7-8e8e-1f7da989149f\" (UID: \"e803824d-f79b-40d7-8e8e-1f7da989149f\") "
Nov 04 00:09:20.944896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.944864    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca" pod "e803824d-f79b-40d7-8e8e-1f7da989149f" (UID: "e803824d-f79b-40d7-8e8e-1f7da989149f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:20.946439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.946402    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7" pod "e803824d-f79b-40d7-8e8e-1f7da989149f" (UID: "e803824d-f79b-40d7-8e8e-1f7da989149f"). InnerVolumeSpecName "kube-api-access-bw6t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:20.948154 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:20.948123    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca" pod "e803824d-f79b-40d7-8e8e-1f7da989149f" (UID: "e803824d-f79b-40d7-8e8e-1f7da989149f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:09:21.045526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:21.045492    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw6t7\" (UniqueName: \"kubernetes.io/projected/e803824d-f79b-40d7-8e8e-1f7da989149f-kube-api-access-bw6t7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:21.045526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:21.045522    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:21.045720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:21.045534    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803824d-f79b-40d7-8e8e-1f7da989149f-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:21.668083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:21.668054    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"
Nov 04 00:09:21.668242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:21.668089    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz" event={"ID":"e803824d-f79b-40d7-8e8e-1f7da989149f","Type":"ContainerDied","Data":"af4b9fe5d51dd0e91981ad62e1d8358bfb846ced8e613baff61daf69fdbf2846"}
Nov 04 00:09:21.668242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:21.668140    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4b9fe5d51dd0e91981ad62e1d8358bfb846ced8e613baff61daf69fdbf2846"
Nov 04 00:09:26.620524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:26.620493    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:27.633423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:27.633391    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:09:27.714401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:27.714358    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"]
Nov 04 00:09:28.036100 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:28.036056    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:09:28.257350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.257310    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"]
Nov 04 00:09:28.258169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.258152    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e803824d-f79b-40d7-8e8e-1f7da989149f" containerName="revoke-keys"
Nov 04 00:09:28.258169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.258170    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e803824d-f79b-40d7-8e8e-1f7da989149f" containerName="revoke-keys"
Nov 04 00:09:28.258306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.258297    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e803824d-f79b-40d7-8e8e-1f7da989149f" containerName="revoke-keys"
Nov 04 00:09:28.265016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.264993    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.273074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.273049    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"]
Nov 04 00:09:28.273371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.273348    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"payload-processing-external-model-e2e--5c4b218e-dockercfg-mj9dh\""
Nov 04 00:09:28.273371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.273356    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"agc-cred-4f59a4a9da80\""
Nov 04 00:09:28.273561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.273541    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"routing-overlay\""
Nov 04 00:09:28.273627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.273609    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-71b2f3\""
Nov 04 00:09:28.276559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.276534    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"openshift-service-ca.crt\""
Nov 04 00:09:28.276657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.276565    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"kube-root-ca.crt\""
Nov 04 00:09:28.370527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.370451    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.370663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.370576    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.370701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.370682    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.370786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.370765    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhlr\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.444883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.444847    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"]
Nov 04 00:09:28.461329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.461286    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.464604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.462500    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"]
Nov 04 00:09:28.469757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.466591    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"agc-cred-22b6782a372e\""
Nov 04 00:09:28.480666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.472569    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.480666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.472677    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.480666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.472855    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.480666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.472887    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhlr\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.480666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.473999    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.480666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.476361    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.510870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.501575    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.510870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.504593    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhlr\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.574651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.574517    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp68s\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.574651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.574641    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.574897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.574843    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.574989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.574973    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.584123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.584098    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:28.678138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.677303    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.678138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.677449    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.678138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.677594    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp68s\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.678138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.677641    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.679851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.679817    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.679989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.679956    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.682339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.681313    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.695281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.695257    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp68s\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.852698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.852657    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:28.879799 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:09:28.879734    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca12f913_b4a1_41e3_8d93_7058398a4bc7.slice/crio-263b99abe8d77139f8bb8cdc66fb88cfe9d4a3ecbd9b2d93c3cde8e11a6d0f24 WatchSource:0}: Error finding container 263b99abe8d77139f8bb8cdc66fb88cfe9d4a3ecbd9b2d93c3cde8e11a6d0f24: Status 404 returned error can't find the container with id 263b99abe8d77139f8bb8cdc66fb88cfe9d4a3ecbd9b2d93c3cde8e11a6d0f24
Nov 04 00:09:28.896874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:28.896843    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"]
Nov 04 00:09:29.151303 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:09:29.151236    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b75c15a_d438_475f_a9df_88a3df3cd5e7.slice/crio-0a245cc18bf1c2d6a9b568a78eb61cf6d3a120151af36f5e166d2e02bd783c69 WatchSource:0}: Error finding container 0a245cc18bf1c2d6a9b568a78eb61cf6d3a120151af36f5e166d2e02bd783c69: Status 404 returned error can't find the container with id 0a245cc18bf1c2d6a9b568a78eb61cf6d3a120151af36f5e166d2e02bd783c69
Nov 04 00:09:29.152298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.152176    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"]
Nov 04 00:09:29.769336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.769297    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" event={"ID":"ca12f913-b4a1-41e3-8d93-7058398a4bc7","Type":"ContainerStarted","Data":"228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8"}
Nov 04 00:09:29.769831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.769366    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" event={"ID":"ca12f913-b4a1-41e3-8d93-7058398a4bc7","Type":"ContainerStarted","Data":"263b99abe8d77139f8bb8cdc66fb88cfe9d4a3ecbd9b2d93c3cde8e11a6d0f24"}
Nov 04 00:09:29.769831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.769627    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:29.774527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.772572    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" event={"ID":"8b75c15a-d438-475f-a9df-88a3df3cd5e7","Type":"ContainerStarted","Data":"c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990"}
Nov 04 00:09:29.774527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.772617    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" event={"ID":"8b75c15a-d438-475f-a9df-88a3df3cd5e7","Type":"ContainerStarted","Data":"0a245cc18bf1c2d6a9b568a78eb61cf6d3a120151af36f5e166d2e02bd783c69"}
Nov 04 00:09:29.774527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.773363    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:29.810761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.810074    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" podStartSLOduration=1.810058564 podStartE2EDuration="1.810058564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:29.808006072 +0000 UTC m=+2434.907606242" watchObservedRunningTime="2026-11-04 00:09:29.810058564 +0000 UTC m=+2434.909658713"
Nov 04 00:09:29.844071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:29.843729    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" podStartSLOduration=1.843710101 podStartE2EDuration="1.843710101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:29.841212259 +0000 UTC m=+2434.940812404" watchObservedRunningTime="2026-11-04 00:09:29.843710101 +0000 UTC m=+2434.943310245"
Nov 04 00:09:30.226561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:30.226527    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"]
Nov 04 00:09:30.533820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:30.529249    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"]
Nov 04 00:09:30.787348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:30.787261    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" secret="" err="secret \"payload-processing-external-model-e2e--5c4b218e-dockercfg-mj9dh\" not found"
Nov 04 00:09:30.918257 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918226    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-22b6782a372e: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:30.918257 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918258    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:30.918520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918274    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-4f59a4a9da80: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:30.918520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918295    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:30.918520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918363    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials podName:8b75c15a-d438-475f-a9df-88a3df3cd5e7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:31.418344783 +0000 UTC m=+2436.517944910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7") : secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:30.918520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918386    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:30.918520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918452    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:31.418433887 +0000 UTC m=+2436.518034029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:30.918520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:30.918493    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:31.418485761 +0000 UTC m=+2436.518085922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:31.210795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:31.210450    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"]
Nov 04 00:09:31.425871 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.425824    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.425925    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:32.425907225 +0000 UTC m=+2437.525507350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.425953    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-22b6782a372e: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.425977    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-4f59a4a9da80: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.425999    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.426047    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:32.426032263 +0000 UTC m=+2437.525632402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.425981    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:31.426088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:31.426089    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials podName:8b75c15a-d438-475f-a9df-88a3df3cd5e7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:32.42608112 +0000 UTC m=+2437.525681299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7") : secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:32.426760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.426206    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"]
Nov 04 00:09:32.445804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.433997    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.446295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.446187    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"]
Nov 04 00:09:32.454883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.454817    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-e2e--7c418871-dockercfg-q8bqm\""
Nov 04 00:09:32.455465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.455440    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"agc-cred-1f4ceb1661c7\""
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.459502    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.459573    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:34.459553835 +0000 UTC m=+2439.559153972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.460017    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-22b6782a372e: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.460039    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.460088    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials podName:8b75c15a-d438-475f-a9df-88a3df3cd5e7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:34.460071899 +0000 UTC m=+2439.559672024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7") : secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.460152    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-4f59a4a9da80: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.460163    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:32.460216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:32.460194    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:34.460183805 +0000 UTC m=+2439.559783929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:32.560949 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.560902    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.561143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.561048    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hsl\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.561464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.561441    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.561507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.561491    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.663174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.663132    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.663354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.663200    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.663354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.663308    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.663504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.663367    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hsl\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.664171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.664134    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.664171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.664142    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.668253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.668226    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.701393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.701354    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hsl\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:32.727791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.716067    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d" podUID="94568fc7-11ca-4918-b7a7-37b89ef96227" containerName="payload-processing" containerID="cri-o://72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c" gracePeriod=25
Nov 04 00:09:32.796424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:32.796383    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:33.081194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.081149    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:33.152761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.152710    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"]
Nov 04 00:09:33.180342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.180267    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials\") pod \"94568fc7-11ca-4918-b7a7-37b89ef96227\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") "
Nov 04 00:09:33.180527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.180353    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume\") pod \"94568fc7-11ca-4918-b7a7-37b89ef96227\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") "
Nov 04 00:09:33.180593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.180532    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume\") pod \"94568fc7-11ca-4918-b7a7-37b89ef96227\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") "
Nov 04 00:09:33.180680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.180659    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689\") pod \"94568fc7-11ca-4918-b7a7-37b89ef96227\" (UID: \"94568fc7-11ca-4918-b7a7-37b89ef96227\") "
Nov 04 00:09:33.181695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.181438    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume" pod "94568fc7-11ca-4918-b7a7-37b89ef96227" (UID: "94568fc7-11ca-4918-b7a7-37b89ef96227"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:33.181695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.181660    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:33.184905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.184878    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials" pod "94568fc7-11ca-4918-b7a7-37b89ef96227" (UID: "94568fc7-11ca-4918-b7a7-37b89ef96227"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:33.186006 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.185972    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume" pod "94568fc7-11ca-4918-b7a7-37b89ef96227" (UID: "94568fc7-11ca-4918-b7a7-37b89ef96227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:33.186137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.186066    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689" pod "94568fc7-11ca-4918-b7a7-37b89ef96227" (UID: "94568fc7-11ca-4918-b7a7-37b89ef96227"). InnerVolumeSpecName "kube-api-access-tt689". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:33.284134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.284093    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt689\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-kube-api-access-tt689\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:33.284134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.284132    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94568fc7-11ca-4918-b7a7-37b89ef96227-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:33.284362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.284148    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94568fc7-11ca-4918-b7a7-37b89ef96227-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:33.627131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.627093    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"]
Nov 04 00:09:33.627983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.627965    5074 cpu_manager.go:414] "containerMap: removing container" podUID="94568fc7-11ca-4918-b7a7-37b89ef96227" containerName="payload-processing"
Nov 04 00:09:33.627983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.627983    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94568fc7-11ca-4918-b7a7-37b89ef96227" containerName="payload-processing"
Nov 04 00:09:33.628161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.628151    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="94568fc7-11ca-4918-b7a7-37b89ef96227" containerName="payload-processing"
Nov 04 00:09:33.632236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.632210    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.694020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.691213    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"]
Nov 04 00:09:33.697127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.697105    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"agc-cred-6e445ec625ea\""
Nov 04 00:09:33.794451 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.794412    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.794656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.794463    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.794656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.794565    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ctk\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.794656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.794598    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.813846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.813812    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw" event={"ID":"467c9e4a-881a-4439-9633-bbd3047df4c6","Type":"ContainerStarted","Data":"b117e3154faa03aeeb41ca74509ffbb355cc5e09d976b2d3dba52f444b298e8a"}
Nov 04 00:09:33.814033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.813887    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw" event={"ID":"467c9e4a-881a-4439-9633-bbd3047df4c6","Type":"ContainerStarted","Data":"92f801befe8f6d59446237f68c0a75a3ec3f98408ba2e304bdfd434668e58ce5"}
Nov 04 00:09:33.814033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.814011    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:33.815418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.815392    5074 generic.go:356] "Generic (PLEG): container finished" podID="94568fc7-11ca-4918-b7a7-37b89ef96227" containerID="72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c" exitCode=0
Nov 04 00:09:33.815511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.815437    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"
Nov 04 00:09:33.815511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.815477    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d" event={"ID":"94568fc7-11ca-4918-b7a7-37b89ef96227","Type":"ContainerDied","Data":"72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c"}
Nov 04 00:09:33.815593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.815525    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d" event={"ID":"94568fc7-11ca-4918-b7a7-37b89ef96227","Type":"ContainerDied","Data":"c2b5bfbec709e03f5db9fafd69db728beaf50d3b68344aa11c91408148023b81"}
Nov 04 00:09:33.815593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.815553    5074 scope.go:122] "RemoveContainer" containerID="72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c"
Nov 04 00:09:33.827194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.827175    5074 scope.go:122] "RemoveContainer" containerID="72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c"
Nov 04 00:09:33.827503 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:33.827477    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c\": container with ID starting with 72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c not found: ID does not exist" containerID="72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c"
Nov 04 00:09:33.827632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.827508    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c"} err="failed to get container status \"72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c\": rpc error: code = NotFound desc = could not find container \"72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c\": container with ID starting with 72527befcbab2da2864f5cf305794cd84c8e1e04ed4374f806d04341b3d1373c not found: ID does not exist"
Nov 04 00:09:33.859345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.859292    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw" podStartSLOduration=1.859276572 podStartE2EDuration="1.859276572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:33.85598996 +0000 UTC m=+2438.955590104" watchObservedRunningTime="2026-11-04 00:09:33.859276572 +0000 UTC m=+2438.958876717"
Nov 04 00:09:33.895979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.895911    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.895979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.895974    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.896194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.896010    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ctk\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.896194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.896032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.896923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.896899    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.897052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.897025    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.898657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.898636    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.906725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.906698    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"]
Nov 04 00:09:33.913231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.913197    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ctk\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:33.916073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.916052    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cmw64d"]
Nov 04 00:09:33.953608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:33.953578    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:34.251452 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:09:34.251421    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d232e3f_8f8c_4ac1_a3ca_2895bf5fdabb.slice/crio-2e6d90b8787ae6fb4cba94fd0693ee30b6d0bccf6d01f894c30a6104fca74505 WatchSource:0}: Error finding container 2e6d90b8787ae6fb4cba94fd0693ee30b6d0bccf6d01f894c30a6104fca74505: Status 404 returned error can't find the container with id 2e6d90b8787ae6fb4cba94fd0693ee30b6d0bccf6d01f894c30a6104fca74505
Nov 04 00:09:34.259231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:34.259201    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"]
Nov 04 00:09:34.506817 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506727    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-22b6782a372e: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:34.506817 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506765    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-4f59a4a9da80: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:34.506817 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506797    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb: secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:34.507061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506767    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:34.507061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506854    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:38.506836084 +0000 UTC m=+2443.606436220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : secret "agc-cred-4f59a4a9da80" not found
Nov 04 00:09:34.507061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506874    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume podName:ca12f913-b4a1-41e3-8d93-7058398a4bc7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:38.506863218 +0000 UTC m=+2443.606463340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:09:34.507061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506770    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d: secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:34.507061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:34.506915    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials podName:8b75c15a-d438-475f-a9df-88a3df3cd5e7 nodeName:}" failed. No retries permitted until 2026-11-04 00:09:38.506907435 +0000 UTC m=+2443.606507557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7") : secret "agc-cred-22b6782a372e" not found
Nov 04 00:09:34.826905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:34.826812    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c" event={"ID":"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb","Type":"ContainerStarted","Data":"622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167"}
Nov 04 00:09:34.826905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:34.826876    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c" event={"ID":"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb","Type":"ContainerStarted","Data":"2e6d90b8787ae6fb4cba94fd0693ee30b6d0bccf6d01f894c30a6104fca74505"}
Nov 04 00:09:34.827330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:34.827060    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:34.868292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:34.868230    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c" podStartSLOduration=1.8682127099999999 podStartE2EDuration="1.86821271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:09:34.856391083 +0000 UTC m=+2439.955991224" watchObservedRunningTime="2026-11-04 00:09:34.86821271 +0000 UTC m=+2439.967812855"
Nov 04 00:09:35.020569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.020533    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94568fc7-11ca-4918-b7a7-37b89ef96227" path="/var/lib/kubelet/pods/94568fc7-11ca-4918-b7a7-37b89ef96227/volumes"
Nov 04 00:09:35.227471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.227432    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs" podUID="de7532bb-68ce-4fbd-a22f-a242315d022b" containerName="payload-processing" containerID="cri-o://6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633" gracePeriod=25
Nov 04 00:09:35.533564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.533533    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:09:35.628758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.628720    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume\") pod \"de7532bb-68ce-4fbd-a22f-a242315d022b\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") "
Nov 04 00:09:35.628939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.628796    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume\") pod \"de7532bb-68ce-4fbd-a22f-a242315d022b\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") "
Nov 04 00:09:35.628939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.628853    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t\") pod \"de7532bb-68ce-4fbd-a22f-a242315d022b\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") "
Nov 04 00:09:35.629046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.628963    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials\") pod \"de7532bb-68ce-4fbd-a22f-a242315d022b\" (UID: \"de7532bb-68ce-4fbd-a22f-a242315d022b\") "
Nov 04 00:09:35.629351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.629320    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume" pod "de7532bb-68ce-4fbd-a22f-a242315d022b" (UID: "de7532bb-68ce-4fbd-a22f-a242315d022b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:35.629566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.629326    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume" pod "de7532bb-68ce-4fbd-a22f-a242315d022b" (UID: "de7532bb-68ce-4fbd-a22f-a242315d022b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:35.629905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.629882    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:35.629905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.629905    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7532bb-68ce-4fbd-a22f-a242315d022b-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:35.631612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.631584    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials" pod "de7532bb-68ce-4fbd-a22f-a242315d022b" (UID: "de7532bb-68ce-4fbd-a22f-a242315d022b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:35.631721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.631625    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t" pod "de7532bb-68ce-4fbd-a22f-a242315d022b" (UID: "de7532bb-68ce-4fbd-a22f-a242315d022b"). InnerVolumeSpecName "kube-api-access-ddv6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:35.731279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.731243    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddv6t\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-kube-api-access-ddv6t\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:35.731279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.731273    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7532bb-68ce-4fbd-a22f-a242315d022b-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:35.838086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.838004    5074 generic.go:356] "Generic (PLEG): container finished" podID="de7532bb-68ce-4fbd-a22f-a242315d022b" containerID="6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633" exitCode=0
Nov 04 00:09:35.838086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.838049    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"
Nov 04 00:09:35.838660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.838089    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs" event={"ID":"de7532bb-68ce-4fbd-a22f-a242315d022b","Type":"ContainerDied","Data":"6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633"}
Nov 04 00:09:35.838660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.838165    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs" event={"ID":"de7532bb-68ce-4fbd-a22f-a242315d022b","Type":"ContainerDied","Data":"ee6c1c9be41a19bc6aeb24cd904090bca84ac38e99883592aa2bc3cdc55e6b3a"}
Nov 04 00:09:35.838660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.838188    5074 scope.go:122] "RemoveContainer" containerID="6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633"
Nov 04 00:09:35.851180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.851158    5074 scope.go:122] "RemoveContainer" containerID="6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633"
Nov 04 00:09:35.851473 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:35.851451    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633\": container with ID starting with 6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633 not found: ID does not exist" containerID="6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633"
Nov 04 00:09:35.851568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.851478    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633"} err="failed to get container status \"6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633\": rpc error: code = NotFound desc = could not find container \"6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633\": container with ID starting with 6d0b83ae822514267ec043550297e45dfb84eba136c72c9823561aad8908d633 not found: ID does not exist"
Nov 04 00:09:35.904353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.904328    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"]
Nov 04 00:09:35.923989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:35.923966    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-76778mlfs"]
Nov 04 00:09:36.794698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:36.794655    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" podUID="ca12f913-b4a1-41e3-8d93-7058398a4bc7" containerName="payload-processing" containerID="cri-o://228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8" gracePeriod=25
Nov 04 00:09:37.027048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.027010    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7532bb-68ce-4fbd-a22f-a242315d022b" path="/var/lib/kubelet/pods/de7532bb-68ce-4fbd-a22f-a242315d022b/volumes"
Nov 04 00:09:37.082624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.082599    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:37.253646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.253608    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials\") pod \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") "
Nov 04 00:09:37.253882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.253673    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume\") pod \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") "
Nov 04 00:09:37.253882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.253874    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr\") pod \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") "
Nov 04 00:09:37.254089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.254068    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume\") pod \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\" (UID: \"ca12f913-b4a1-41e3-8d93-7058398a4bc7\") "
Nov 04 00:09:37.254089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.254079    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume" pod "ca12f913-b4a1-41e3-8d93-7058398a4bc7" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:37.254528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.254505    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume" pod "ca12f913-b4a1-41e3-8d93-7058398a4bc7" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:37.254704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.254686    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:37.254812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.254705    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca12f913-b4a1-41e3-8d93-7058398a4bc7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:37.256187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.256164    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials" pod "ca12f913-b4a1-41e3-8d93-7058398a4bc7" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:37.256289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.256198    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr" pod "ca12f913-b4a1-41e3-8d93-7058398a4bc7" (UID: "ca12f913-b4a1-41e3-8d93-7058398a4bc7"). InnerVolumeSpecName "kube-api-access-rfhlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:37.356437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.356351    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:37.356437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.356380    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfhlr\" (UniqueName: \"kubernetes.io/projected/ca12f913-b4a1-41e3-8d93-7058398a4bc7-kube-api-access-rfhlr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:37.805285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.805247    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" podUID="8b75c15a-d438-475f-a9df-88a3df3cd5e7" containerName="payload-processing" containerID="cri-o://c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990" gracePeriod=25
Nov 04 00:09:37.859014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.858982    5074 generic.go:356] "Generic (PLEG): container finished" podID="ca12f913-b4a1-41e3-8d93-7058398a4bc7" containerID="228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8" exitCode=0
Nov 04 00:09:37.859131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.859029    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"
Nov 04 00:09:37.859131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.859071    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" event={"ID":"ca12f913-b4a1-41e3-8d93-7058398a4bc7","Type":"ContainerDied","Data":"228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8"}
Nov 04 00:09:37.859213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.859145    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb" event={"ID":"ca12f913-b4a1-41e3-8d93-7058398a4bc7","Type":"ContainerDied","Data":"263b99abe8d77139f8bb8cdc66fb88cfe9d4a3ecbd9b2d93c3cde8e11a6d0f24"}
Nov 04 00:09:37.859213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.859178    5074 scope.go:122] "RemoveContainer" containerID="228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8"
Nov 04 00:09:37.918434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.918408    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"]
Nov 04 00:09:37.920160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.920141    5074 scope.go:122] "RemoveContainer" containerID="228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8"
Nov 04 00:09:37.920496 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:37.920474    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8\": container with ID starting with 228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8 not found: ID does not exist" containerID="228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8"
Nov 04 00:09:37.920563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.920506    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8"} err="failed to get container status \"228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8\": rpc error: code = NotFound desc = could not find container \"228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8\": container with ID starting with 228379c166c4709b4a361baf76b6eb3fdad65bfda1ce3ec94d88fa8dc2fe06b8 not found: ID does not exist"
Nov 04 00:09:37.927229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:37.927201    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-67d6bnxbb"]
Nov 04 00:09:38.089264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.089228    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:38.268859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.268825    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume\") pod \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") "
Nov 04 00:09:38.269038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.268964    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials\") pod \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") "
Nov 04 00:09:38.269038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.269013    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s\") pod \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") "
Nov 04 00:09:38.269176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.269061    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume\") pod \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\" (UID: \"8b75c15a-d438-475f-a9df-88a3df3cd5e7\") "
Nov 04 00:09:38.269323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.269298    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume" pod "8b75c15a-d438-475f-a9df-88a3df3cd5e7" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:38.269521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.269495    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume" pod "8b75c15a-d438-475f-a9df-88a3df3cd5e7" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:38.269679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.269658    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:38.269679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.269678    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b75c15a-d438-475f-a9df-88a3df3cd5e7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:38.271461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.271434    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s" pod "8b75c15a-d438-475f-a9df-88a3df3cd5e7" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7"). InnerVolumeSpecName "kube-api-access-gp68s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:38.271547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.271471    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials" pod "8b75c15a-d438-475f-a9df-88a3df3cd5e7" (UID: "8b75c15a-d438-475f-a9df-88a3df3cd5e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:38.370671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.370596    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:38.370671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.370629    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp68s\" (UniqueName: \"kubernetes.io/projected/8b75c15a-d438-475f-a9df-88a3df3cd5e7-kube-api-access-gp68s\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:38.869976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.869938    5074 generic.go:356] "Generic (PLEG): container finished" podID="8b75c15a-d438-475f-a9df-88a3df3cd5e7" containerID="c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990" exitCode=0
Nov 04 00:09:38.870170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.869979    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" event={"ID":"8b75c15a-d438-475f-a9df-88a3df3cd5e7","Type":"ContainerDied","Data":"c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990"}
Nov 04 00:09:38.870170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.870002    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"
Nov 04 00:09:38.870170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.870038    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d" event={"ID":"8b75c15a-d438-475f-a9df-88a3df3cd5e7","Type":"ContainerDied","Data":"0a245cc18bf1c2d6a9b568a78eb61cf6d3a120151af36f5e166d2e02bd783c69"}
Nov 04 00:09:38.870170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.870067    5074 scope.go:122] "RemoveContainer" containerID="c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990"
Nov 04 00:09:38.883104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.883081    5074 scope.go:122] "RemoveContainer" containerID="c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990"
Nov 04 00:09:38.883389 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:38.883366    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990\": container with ID starting with c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990 not found: ID does not exist" containerID="c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990"
Nov 04 00:09:38.883462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.883397    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990"} err="failed to get container status \"c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990\": rpc error: code = NotFound desc = could not find container \"c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990\": container with ID starting with c06d993fc338ee8c7b20f70821e23756d7bed3f5d3cd5816d18001bee3c60990 not found: ID does not exist"
Nov 04 00:09:38.902511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:38.902481    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"]
Nov 04 00:09:39.020758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:39.020707    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca12f913-b4a1-41e3-8d93-7058398a4bc7" path="/var/lib/kubelet/pods/ca12f913-b4a1-41e3-8d93-7058398a4bc7/volumes"
Nov 04 00:09:39.021117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:39.021099    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7877bcq7d"]
Nov 04 00:09:41.024710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:41.024667    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b75c15a-d438-475f-a9df-88a3df3cd5e7" path="/var/lib/kubelet/pods/8b75c15a-d438-475f-a9df-88a3df3cd5e7/volumes"
Nov 04 00:09:44.829646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:44.829611    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:09:45.843435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:45.843398    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:09:45.916780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:45.916701    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"]
Nov 04 00:09:49.247082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:49.247032    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"]
Nov 04 00:09:49.247871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:49.247309    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy" containerID="cri-o://1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d" gracePeriod=30
Nov 04 00:09:50.917814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:50.917773    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc" podUID="619acd5c-922a-45a3-a670-896ca43bb518" containerName="payload-processing" containerID="cri-o://81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405" gracePeriod=25
Nov 04 00:09:51.219180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.219118    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:09:51.341464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.341425    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps\") pod \"619acd5c-922a-45a3-a670-896ca43bb518\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") "
Nov 04 00:09:51.341669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.341584    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials\") pod \"619acd5c-922a-45a3-a670-896ca43bb518\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") "
Nov 04 00:09:51.341669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.341632    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume\") pod \"619acd5c-922a-45a3-a670-896ca43bb518\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") "
Nov 04 00:09:51.341853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.341831    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume\") pod \"619acd5c-922a-45a3-a670-896ca43bb518\" (UID: \"619acd5c-922a-45a3-a670-896ca43bb518\") "
Nov 04 00:09:51.342202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.342154    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume" pod "619acd5c-922a-45a3-a670-896ca43bb518" (UID: "619acd5c-922a-45a3-a670-896ca43bb518"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:51.342312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.342247    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume" pod "619acd5c-922a-45a3-a670-896ca43bb518" (UID: "619acd5c-922a-45a3-a670-896ca43bb518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:51.342422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.342407    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:51.342469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.342423    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619acd5c-922a-45a3-a670-896ca43bb518-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:51.344124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.344098    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials" pod "619acd5c-922a-45a3-a670-896ca43bb518" (UID: "619acd5c-922a-45a3-a670-896ca43bb518"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:51.344255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.344228    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps" pod "619acd5c-922a-45a3-a670-896ca43bb518" (UID: "619acd5c-922a-45a3-a670-896ca43bb518"). InnerVolumeSpecName "kube-api-access-ff2ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:51.443548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.443516    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2ps\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-kube-api-access-ff2ps\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:51.443548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.443545    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/619acd5c-922a-45a3-a670-896ca43bb518-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:51.995359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.995319    5074 generic.go:356] "Generic (PLEG): container finished" podID="619acd5c-922a-45a3-a670-896ca43bb518" containerID="81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405" exitCode=0
Nov 04 00:09:51.995874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.995368    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"
Nov 04 00:09:51.995874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.995377    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc" event={"ID":"619acd5c-922a-45a3-a670-896ca43bb518","Type":"ContainerDied","Data":"81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405"}
Nov 04 00:09:51.995874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.995505    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc" event={"ID":"619acd5c-922a-45a3-a670-896ca43bb518","Type":"ContainerDied","Data":"658497f18357fb2bb277a118b9edaf652bc5b186a6901ca3fa30c561cb995b25"}
Nov 04 00:09:51.995874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:51.995533    5074 scope.go:122] "RemoveContainer" containerID="81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405"
Nov 04 00:09:52.010734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:52.010706    5074 scope.go:122] "RemoveContainer" containerID="81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405"
Nov 04 00:09:52.011063 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:52.011038    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405\": container with ID starting with 81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405 not found: ID does not exist" containerID="81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405"
Nov 04 00:09:52.011141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:52.011074    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405"} err="failed to get container status \"81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405\": rpc error: code = NotFound desc = could not find container \"81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405\": container with ID starting with 81302b4f04ddda7c82fbd96ec758d3c3bfe8a9dd9e8d5c43a97b16bb841c0405 not found: ID does not exist"
Nov 04 00:09:52.031627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:52.031595    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"]
Nov 04 00:09:52.045455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:52.045426    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fglgkc"]
Nov 04 00:09:53.019274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:53.019235    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619acd5c-922a-45a3-a670-896ca43bb518" path="/var/lib/kubelet/pods/619acd5c-922a-45a3-a670-896ca43bb518/volumes"
Nov 04 00:09:54.544962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.544880    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 04 00:09:54.688920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.688879    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.688982    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689046    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689103    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689135    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689226    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689309    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689391    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.689521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.689501    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket\") pod \"c3120e42-afef-4b73-812f-85c2614e398c\" (UID: \"c3120e42-afef-4b73-812f-85c2614e398c\") "
Nov 04 00:09:54.690320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.690294    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:54.690925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.690901    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:54.691058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.690916    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:54.691058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.690947    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:54.691157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.690987    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:09:54.692498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.692474    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:54.692618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.692601    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "kube-api-access-5w58b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:09:54.693069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.693042    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:09:54.693069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.693058    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy" pod "c3120e42-afef-4b73-812f-85c2614e398c" (UID: "c3120e42-afef-4b73-812f-85c2614e398c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:09:54.791762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791700    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791733    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791787    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791797    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791808    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w58b\" (UniqueName: \"kubernetes.io/projected/c3120e42-afef-4b73-812f-85c2614e398c-kube-api-access-5w58b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791818    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791827    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3120e42-afef-4b73-812f-85c2614e398c-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791835    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3120e42-afef-4b73-812f-85c2614e398c-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:54.791933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:54.791844    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3120e42-afef-4b73-812f-85c2614e398c-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:09:55.027681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.027642    5074 generic.go:356] "Generic (PLEG): container finished" podID="c3120e42-afef-4b73-812f-85c2614e398c" containerID="1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d" exitCode=0
Nov 04 00:09:55.027894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.027702    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"
Nov 04 00:09:55.027894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.027726    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" event={"ID":"c3120e42-afef-4b73-812f-85c2614e398c","Type":"ContainerDied","Data":"1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d"}
Nov 04 00:09:55.027894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.027802    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl" event={"ID":"c3120e42-afef-4b73-812f-85c2614e398c","Type":"ContainerDied","Data":"580a4ab3a7738d8fb62bb2ef22eaa61d42c38b2908d89a8556fc61e44f8567fd"}
Nov 04 00:09:55.027894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.027834    5074 scope.go:122] "RemoveContainer" containerID="1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d"
Nov 04 00:09:55.046599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.046566    5074 scope.go:122] "RemoveContainer" containerID="1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d"
Nov 04 00:09:55.046988 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:55.046963    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d\": container with ID starting with 1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d not found: ID does not exist" containerID="1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d"
Nov 04 00:09:55.047081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.046999    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d"} err="failed to get container status \"1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d\": rpc error: code = NotFound desc = could not find container \"1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d\": container with ID starting with 1abbb4454e7d16e38e1a3f7c502d045ebf586daa23209a5d93e97a69ee5a617d not found: ID does not exist"
Nov 04 00:09:55.065413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.065389    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"]
Nov 04 00:09:55.078400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:55.078375    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sxqnl"]
Nov 04 00:09:57.020026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:57.019988    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3120e42-afef-4b73-812f-85c2614e398c" path="/var/lib/kubelet/pods/c3120e42-afef-4b73-812f-85c2614e398c/volumes"
Nov 04 00:09:58.025951 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:09:58.025905    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:09:59.154267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:09:59.154231    5074 scope.go:122] "RemoveContainer" containerID="2df3083dfa6fe33e5e73100dd50035c13c6da7b3a3f07612df1702603ac1022a"
Nov 04 00:10:04.624101 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624069    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"]
Nov 04 00:10:04.624875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624858    5074 cpu_manager.go:414] "containerMap: removing container" podUID="619acd5c-922a-45a3-a670-896ca43bb518" containerName="payload-processing"
Nov 04 00:10:04.624946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624876    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619acd5c-922a-45a3-a670-896ca43bb518" containerName="payload-processing"
Nov 04 00:10:04.624946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624891    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8b75c15a-d438-475f-a9df-88a3df3cd5e7" containerName="payload-processing"
Nov 04 00:10:04.624946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624897    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b75c15a-d438-475f-a9df-88a3df3cd5e7" containerName="payload-processing"
Nov 04 00:10:04.624946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624929    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ca12f913-b4a1-41e3-8d93-7058398a4bc7" containerName="payload-processing"
Nov 04 00:10:04.624946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624934    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca12f913-b4a1-41e3-8d93-7058398a4bc7" containerName="payload-processing"
Nov 04 00:10:04.624946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624944    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624950    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624965    5074 cpu_manager.go:414] "containerMap: removing container" podUID="de7532bb-68ce-4fbd-a22f-a242315d022b" containerName="payload-processing"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.624970    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7532bb-68ce-4fbd-a22f-a242315d022b" containerName="payload-processing"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.625068    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca12f913-b4a1-41e3-8d93-7058398a4bc7" containerName="payload-processing"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.625078    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3120e42-afef-4b73-812f-85c2614e398c" containerName="istio-proxy"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.625086    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7532bb-68ce-4fbd-a22f-a242315d022b" containerName="payload-processing"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.625094    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="619acd5c-922a-45a3-a670-896ca43bb518" containerName="payload-processing"
Nov 04 00:10:04.625149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.625102    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b75c15a-d438-475f-a9df-88a3df3cd5e7" containerName="payload-processing"
Nov 04 00:10:04.640044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.640015    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.648604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.648570    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"]
Nov 04 00:10:04.649379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.649356    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:10:04.652875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.652850    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:10:04.722065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.722031    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhl7m\" (UniqueName: \"kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.722287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.722258    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.722353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.722313    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.824064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.824013    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhl7m\" (UniqueName: \"kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.824287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.824164    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.824287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.824193    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.825054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.825028    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.826787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.826767    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.838066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.838039    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhl7m\" (UniqueName: \"kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m\") pod \"maas-api-revoke-sub-eac8d69a-f2grz\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:04.963026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:04.962989    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:05.256539 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:05.256504    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c51d89_7648_49ae_87fd_f6970459e180.slice/crio-81e658a7605d92e9e0edb108935ee02eb5e1af36f757494c5fed444af13866c0 WatchSource:0}: Error finding container 81e658a7605d92e9e0edb108935ee02eb5e1af36f757494c5fed444af13866c0: Status 404 returned error can't find the container with id 81e658a7605d92e9e0edb108935ee02eb5e1af36f757494c5fed444af13866c0
Nov 04 00:10:05.256893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:05.256863    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"]
Nov 04 00:10:06.123565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:06.123525    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz" event={"ID":"61c51d89-7648-49ae-87fd-f6970459e180","Type":"ContainerStarted","Data":"c3a9f3736311604deba1722152900f752f0d510fcb56b215f5e8c1dcf6a4e76d"}
Nov 04 00:10:06.124058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:06.123596    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz" event={"ID":"61c51d89-7648-49ae-87fd-f6970459e180","Type":"ContainerStarted","Data":"81e658a7605d92e9e0edb108935ee02eb5e1af36f757494c5fed444af13866c0"}
Nov 04 00:10:06.153767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:06.153686    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz" podStartSLOduration=2.153672122 podStartE2EDuration="2.153672122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:06.152317793 +0000 UTC m=+2471.251917938" watchObservedRunningTime="2026-11-04 00:10:06.153672122 +0000 UTC m=+2471.253272293"
Nov 04 00:10:07.133524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:07.133483    5074 generic.go:356] "Generic (PLEG): container finished" podID="61c51d89-7648-49ae-87fd-f6970459e180" containerID="c3a9f3736311604deba1722152900f752f0d510fcb56b215f5e8c1dcf6a4e76d" exitCode=0
Nov 04 00:10:07.134005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:07.133547    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz" event={"ID":"61c51d89-7648-49ae-87fd-f6970459e180","Type":"ContainerDied","Data":"c3a9f3736311604deba1722152900f752f0d510fcb56b215f5e8c1dcf6a4e76d"}
Nov 04 00:10:08.332531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.332500    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:08.367980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.367946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca\") pod \"61c51d89-7648-49ae-87fd-f6970459e180\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") "
Nov 04 00:10:08.368175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.368135    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca\") pod \"61c51d89-7648-49ae-87fd-f6970459e180\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") "
Nov 04 00:10:08.368237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.368193    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m\" (UniqueName: \"kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m\") pod \"61c51d89-7648-49ae-87fd-f6970459e180\" (UID: \"61c51d89-7648-49ae-87fd-f6970459e180\") "
Nov 04 00:10:08.368574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.368545    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca" pod "61c51d89-7648-49ae-87fd-f6970459e180" (UID: "61c51d89-7648-49ae-87fd-f6970459e180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:08.368811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.368791    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:08.370343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.370313    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m" pod "61c51d89-7648-49ae-87fd-f6970459e180" (UID: "61c51d89-7648-49ae-87fd-f6970459e180"). InnerVolumeSpecName "kube-api-access-lhl7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:08.370437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.370338    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca" pod "61c51d89-7648-49ae-87fd-f6970459e180" (UID: "61c51d89-7648-49ae-87fd-f6970459e180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:10:08.469829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.469797    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c51d89-7648-49ae-87fd-f6970459e180-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:08.469829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:08.469832    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhl7m\" (UniqueName: \"kubernetes.io/projected/61c51d89-7648-49ae-87fd-f6970459e180-kube-api-access-lhl7m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:09.153252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:09.153218    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"
Nov 04 00:10:09.153421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:09.153228    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz" event={"ID":"61c51d89-7648-49ae-87fd-f6970459e180","Type":"ContainerDied","Data":"81e658a7605d92e9e0edb108935ee02eb5e1af36f757494c5fed444af13866c0"}
Nov 04 00:10:09.153421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:09.153348    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e658a7605d92e9e0edb108935ee02eb5e1af36f757494c5fed444af13866c0"
Nov 04 00:10:16.550603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.550568    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"]
Nov 04 00:10:16.551367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.551351    5074 cpu_manager.go:414] "containerMap: removing container" podUID="61c51d89-7648-49ae-87fd-f6970459e180" containerName="revoke-keys"
Nov 04 00:10:16.551427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.551368    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c51d89-7648-49ae-87fd-f6970459e180" containerName="revoke-keys"
Nov 04 00:10:16.551532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.551522    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c51d89-7648-49ae-87fd-f6970459e180" containerName="revoke-keys"
Nov 04 00:10:16.555860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.555841    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.564954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.564928    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"]
Nov 04 00:10:16.626903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.626859    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.627094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.626975    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvzd\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.627162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.627129    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.627289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.627270    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.728288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.728249    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.728465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.728398    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.728465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.728447    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvzd\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.728592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.728580    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.729188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.729154    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.729337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.729308    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.731306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.731282    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.742675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.742653    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvzd\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.857946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.857865    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"]
Nov 04 00:10:16.862991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.862972    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:16.872435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.872407    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:10:16.874873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.874813    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:16.875162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.875142    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:10:16.879002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.878966    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"]
Nov 04 00:10:16.931470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.931405    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:16.931652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.931503    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mk28\" (UniqueName: \"kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:16.931652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:16.931577    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.033042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.033003    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.033216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.033185    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mk28\" (UniqueName: \"kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.033362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.033339    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.033996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.033932    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.037440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.037410    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.221093 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:17.221056    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2072bb_1e8d_4eb3_9269_114981cc1d1e.slice/crio-d3bc4fd13bc45aa3c111ad9f9dae30e56bcaab5c4599f273c8d08da182018dab WatchSource:0}: Error finding container d3bc4fd13bc45aa3c111ad9f9dae30e56bcaab5c4599f273c8d08da182018dab: Status 404 returned error can't find the container with id d3bc4fd13bc45aa3c111ad9f9dae30e56bcaab5c4599f273c8d08da182018dab
Nov 04 00:10:17.348803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.348733    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mk28\" (UniqueName: \"kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28\") pod \"maas-api-revoke-sub-8365e2ea-fkc2x\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.359383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.359355    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"]
Nov 04 00:10:17.490240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.490169    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:17.779050 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:17.779013    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0ecb58_3fdb_4562_bc76_c5dd2e481f24.slice/crio-c152faa604b4a3ca663ad83e79450f59e814e685fd76a3a78b7b202c421e79e0 WatchSource:0}: Error finding container c152faa604b4a3ca663ad83e79450f59e814e685fd76a3a78b7b202c421e79e0: Status 404 returned error can't find the container with id c152faa604b4a3ca663ad83e79450f59e814e685fd76a3a78b7b202c421e79e0
Nov 04 00:10:17.784845 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:17.783197    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"]
Nov 04 00:10:18.244991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.244951    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x" event={"ID":"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24","Type":"ContainerStarted","Data":"fc6b404483bc6e43d75ca3684335d650eb310d23a75c6cc59de243e82fdfc2d5"}
Nov 04 00:10:18.245135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.245023    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x" event={"ID":"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24","Type":"ContainerStarted","Data":"c152faa604b4a3ca663ad83e79450f59e814e685fd76a3a78b7b202c421e79e0"}
Nov 04 00:10:18.246894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.246866    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" event={"ID":"2f2072bb-1e8d-4eb3-9269-114981cc1d1e","Type":"ContainerStarted","Data":"545f0550c69deb52f7305864d5f0797740dbeae07a65d467636d06c26851b893"}
Nov 04 00:10:18.247023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.246920    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" event={"ID":"2f2072bb-1e8d-4eb3-9269-114981cc1d1e","Type":"ContainerStarted","Data":"d3bc4fd13bc45aa3c111ad9f9dae30e56bcaab5c4599f273c8d08da182018dab"}
Nov 04 00:10:18.247023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.246964    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:18.311320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.311258    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x" podStartSLOduration=2.31124051 podStartE2EDuration="2.31124051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:18.277466466 +0000 UTC m=+2483.377066612" watchObservedRunningTime="2026-11-04 00:10:18.31124051 +0000 UTC m=+2483.410840656"
Nov 04 00:10:18.316409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:18.316359    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" podStartSLOduration=2.316343676 podStartE2EDuration="2.316343676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:18.304537917 +0000 UTC m=+2483.404138061" watchObservedRunningTime="2026-11-04 00:10:18.316343676 +0000 UTC m=+2483.415943823"
Nov 04 00:10:19.256667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:19.256635    5074 generic.go:356] "Generic (PLEG): container finished" podID="4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" containerID="fc6b404483bc6e43d75ca3684335d650eb310d23a75c6cc59de243e82fdfc2d5" exitCode=0
Nov 04 00:10:19.257112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:19.256723    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x" event={"ID":"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24","Type":"ContainerDied","Data":"fc6b404483bc6e43d75ca3684335d650eb310d23a75c6cc59de243e82fdfc2d5"}
Nov 04 00:10:20.449138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.449108    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:20.476262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.476237    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28\" (UniqueName: \"kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28\") pod \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") "
Nov 04 00:10:20.476388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.476308    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca\") pod \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") "
Nov 04 00:10:20.476388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.476346    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca\") pod \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\" (UID: \"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24\") "
Nov 04 00:10:20.476636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.476614    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca" pod "4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" (UID: "4b0ecb58-3fdb-4562-bc76-c5dd2e481f24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:20.476840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.476823    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:20.478687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.478663    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca" pod "4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" (UID: "4b0ecb58-3fdb-4562-bc76-c5dd2e481f24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:10:20.478687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.478682    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28" pod "4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" (UID: "4b0ecb58-3fdb-4562-bc76-c5dd2e481f24"). InnerVolumeSpecName "kube-api-access-4mk28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:20.577798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.577707    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mk28\" (UniqueName: \"kubernetes.io/projected/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-kube-api-access-4mk28\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:20.577798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:20.577761    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:21.272897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:21.272856    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x" event={"ID":"4b0ecb58-3fdb-4562-bc76-c5dd2e481f24","Type":"ContainerDied","Data":"c152faa604b4a3ca663ad83e79450f59e814e685fd76a3a78b7b202c421e79e0"}
Nov 04 00:10:21.272897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:21.272886    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"
Nov 04 00:10:21.273119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:21.272918    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c152faa604b4a3ca663ad83e79450f59e814e685fd76a3a78b7b202c421e79e0"
Nov 04 00:10:25.545640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.545604    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"]
Nov 04 00:10:25.546550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.546532    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" containerName="revoke-keys"
Nov 04 00:10:25.546598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.546551    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" containerName="revoke-keys"
Nov 04 00:10:25.546759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.546735    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" containerName="revoke-keys"
Nov 04 00:10:25.550655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.550637    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.558210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.558185    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:10:25.559287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.559261    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"]
Nov 04 00:10:25.561133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.560815    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:10:25.645703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.645671    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th29d\" (UniqueName: \"kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.645916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.645837    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.645993 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.645974    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.747309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.747267    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.747476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.747398    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.747531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.747506    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th29d\" (UniqueName: \"kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.748157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.748125    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.749672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.749650    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.761221 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.761201    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th29d\" (UniqueName: \"kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d\") pod \"maas-api-revoke-sub-769e5e62-2mrhd\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:25.871854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:25.871790    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:26.171815 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:26.171775    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc56a90_6844_40fe_9674_8c3e5a05b498.slice/crio-06b42bf37e09fb546321ec5a79e3d6aeb86a6da5903dbfa53ff4e8a1aa891991 WatchSource:0}: Error finding container 06b42bf37e09fb546321ec5a79e3d6aeb86a6da5903dbfa53ff4e8a1aa891991: Status 404 returned error can't find the container with id 06b42bf37e09fb546321ec5a79e3d6aeb86a6da5903dbfa53ff4e8a1aa891991
Nov 04 00:10:26.175524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:26.175494    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"]
Nov 04 00:10:26.327001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:26.326966    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd" event={"ID":"fcc56a90-6844-40fe-9674-8c3e5a05b498","Type":"ContainerStarted","Data":"06b42bf37e09fb546321ec5a79e3d6aeb86a6da5903dbfa53ff4e8a1aa891991"}
Nov 04 00:10:27.336239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:27.336200    5074 generic.go:356] "Generic (PLEG): container finished" podID="fcc56a90-6844-40fe-9674-8c3e5a05b498" containerID="e9ffbc4b2007fb13d694f48bcc4b025d547d148a249b784041955cd6f22d4db1" exitCode=0
Nov 04 00:10:27.336625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:27.336252    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd" event={"ID":"fcc56a90-6844-40fe-9674-8c3e5a05b498","Type":"ContainerDied","Data":"e9ffbc4b2007fb13d694f48bcc4b025d547d148a249b784041955cd6f22d4db1"}
Nov 04 00:10:28.022724 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:28.022689    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:10:28.521920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.521890    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:28.586686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.586645    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d\" (UniqueName: \"kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d\") pod \"fcc56a90-6844-40fe-9674-8c3e5a05b498\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") "
Nov 04 00:10:28.586907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.586860    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca\") pod \"fcc56a90-6844-40fe-9674-8c3e5a05b498\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") "
Nov 04 00:10:28.586987 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.586948    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca\") pod \"fcc56a90-6844-40fe-9674-8c3e5a05b498\" (UID: \"fcc56a90-6844-40fe-9674-8c3e5a05b498\") "
Nov 04 00:10:28.587433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.587405    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca" pod "fcc56a90-6844-40fe-9674-8c3e5a05b498" (UID: "fcc56a90-6844-40fe-9674-8c3e5a05b498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:28.587532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.587514    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:28.589350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.589317    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d" pod "fcc56a90-6844-40fe-9674-8c3e5a05b498" (UID: "fcc56a90-6844-40fe-9674-8c3e5a05b498"). InnerVolumeSpecName "kube-api-access-th29d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:28.589471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.589349    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca" pod "fcc56a90-6844-40fe-9674-8c3e5a05b498" (UID: "fcc56a90-6844-40fe-9674-8c3e5a05b498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:10:28.688542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.688505    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th29d\" (UniqueName: \"kubernetes.io/projected/fcc56a90-6844-40fe-9674-8c3e5a05b498-kube-api-access-th29d\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:28.688542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:28.688537    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc56a90-6844-40fe-9674-8c3e5a05b498-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:29.268919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.268879    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:29.356881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.356791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd" event={"ID":"fcc56a90-6844-40fe-9674-8c3e5a05b498","Type":"ContainerDied","Data":"06b42bf37e09fb546321ec5a79e3d6aeb86a6da5903dbfa53ff4e8a1aa891991"}
Nov 04 00:10:29.356881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.356853    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b42bf37e09fb546321ec5a79e3d6aeb86a6da5903dbfa53ff4e8a1aa891991"
Nov 04 00:10:29.356881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.356881    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"
Nov 04 00:10:29.364571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.364537    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"]
Nov 04 00:10:29.398399 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:29.398370    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-114abfa61a32: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:29.398399 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:29.398402    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:29.398642 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:29.398458    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials podName:2f2072bb-1e8d-4eb3-9269-114981cc1d1e nodeName:}" failed. No retries permitted until 2026-11-04 00:10:29.89843942 +0000 UTC m=+2494.998039548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e") : secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:29.542772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.539474    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"]
Nov 04 00:10:29.542772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.540811    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fcc56a90-6844-40fe-9674-8c3e5a05b498" containerName="revoke-keys"
Nov 04 00:10:29.542772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.540837    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc56a90-6844-40fe-9674-8c3e5a05b498" containerName="revoke-keys"
Nov 04 00:10:29.542772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.541159    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc56a90-6844-40fe-9674-8c3e5a05b498" containerName="revoke-keys"
Nov 04 00:10:29.546925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.546897    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.555118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.555080    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"]
Nov 04 00:10:29.619513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.619428    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.619690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.619604    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.619690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.619668    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jhc\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.619838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.619718    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.721185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.721146    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jhc\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.721371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.721207    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.721430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.721401    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.721574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.721548    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.722286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.722257    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.722461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.722393    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.724041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.724019    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.735328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.735302    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jhc\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.820378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.820335    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"]
Nov 04 00:10:29.826616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.826584    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:29.830792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.830758    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"]
Nov 04 00:10:29.840326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.835403    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:10:29.840326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.838173    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:10:29.869179 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.869147    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:29.924004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.923970    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:29.924173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.924012    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:29.924329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:29.924297    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cr69\" (UniqueName: \"kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:29.924492 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:29.924475    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-114abfa61a32: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:29.924533 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:29.924498    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:29.924573 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:29.924554    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials podName:2f2072bb-1e8d-4eb3-9269-114981cc1d1e nodeName:}" failed. No retries permitted until 2026-11-04 00:10:30.924526675 +0000 UTC m=+2496.024126802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e") : secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:30.026015 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.025963    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cr69\" (UniqueName: \"kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.026184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.026165    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.026227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.026203    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.027658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.027629    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.029408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.029382    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.040927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.040902    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cr69\" (UniqueName: \"kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69\") pod \"maas-api-revoke-sub-c07dc9d0-v7lhf\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.148635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.148564    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:30.162520 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:30.162477    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda858832b_4c68_4e1a_887a_49fe5bf22138.slice/crio-53a5aab0f695f58cc56bc6af4ec70a7b7d1b0e7c18ac808dc0987e206c45f05f WatchSource:0}: Error finding container 53a5aab0f695f58cc56bc6af4ec70a7b7d1b0e7c18ac808dc0987e206c45f05f: Status 404 returned error can't find the container with id 53a5aab0f695f58cc56bc6af4ec70a7b7d1b0e7c18ac808dc0987e206c45f05f
Nov 04 00:10:30.162670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.162654    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"]
Nov 04 00:10:30.366938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.366904    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5" event={"ID":"a858832b-4c68-4e1a-887a-49fe5bf22138","Type":"ContainerStarted","Data":"b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84"}
Nov 04 00:10:30.367132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.366957    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5" event={"ID":"a858832b-4c68-4e1a-887a-49fe5bf22138","Type":"ContainerStarted","Data":"53a5aab0f695f58cc56bc6af4ec70a7b7d1b0e7c18ac808dc0987e206c45f05f"}
Nov 04 00:10:30.367132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.366982    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:30.441151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.441096    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5" podStartSLOduration=1.441082811 podStartE2EDuration="1.441082811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:30.436450042 +0000 UTC m=+2495.536050188" watchObservedRunningTime="2026-11-04 00:10:30.441082811 +0000 UTC m=+2495.540682974"
Nov 04 00:10:30.447350 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:30.447314    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dddeb40_2e71_456a_8fde_a9379b41a963.slice/crio-a435a7b75bf94fe630b5b19e60910fbdf140ec6154d64ecdeb7b6dca460662f5 WatchSource:0}: Error finding container a435a7b75bf94fe630b5b19e60910fbdf140ec6154d64ecdeb7b6dca460662f5: Status 404 returned error can't find the container with id a435a7b75bf94fe630b5b19e60910fbdf140ec6154d64ecdeb7b6dca460662f5
Nov 04 00:10:30.450895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:30.450867    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"]
Nov 04 00:10:30.938726 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:30.938690    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-114abfa61a32: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:30.938726 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:30.938724    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:30.939302 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:30.938811    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials podName:2f2072bb-1e8d-4eb3-9269-114981cc1d1e nodeName:}" failed. No retries permitted until 2026-11-04 00:10:32.938793837 +0000 UTC m=+2498.038393974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e") : secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:31.376063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:31.376018    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf" event={"ID":"1dddeb40-2e71-456a-8fde-a9379b41a963","Type":"ContainerStarted","Data":"1ec74d50dfd4e8376d1fd4332317468824b6d885a62fb1a981aa79ae243a858b"}
Nov 04 00:10:31.376249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:31.376092    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf" event={"ID":"1dddeb40-2e71-456a-8fde-a9379b41a963","Type":"ContainerStarted","Data":"a435a7b75bf94fe630b5b19e60910fbdf140ec6154d64ecdeb7b6dca460662f5"}
Nov 04 00:10:31.405862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:31.405796    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf" podStartSLOduration=2.405778008 podStartE2EDuration="2.405778008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:31.402271516 +0000 UTC m=+2496.501871660" watchObservedRunningTime="2026-11-04 00:10:31.405778008 +0000 UTC m=+2496.505378152"
Nov 04 00:10:32.385527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:32.385489    5074 generic.go:356] "Generic (PLEG): container finished" podID="1dddeb40-2e71-456a-8fde-a9379b41a963" containerID="1ec74d50dfd4e8376d1fd4332317468824b6d885a62fb1a981aa79ae243a858b" exitCode=0
Nov 04 00:10:32.386118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:32.385527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf" event={"ID":"1dddeb40-2e71-456a-8fde-a9379b41a963","Type":"ContainerDied","Data":"1ec74d50dfd4e8376d1fd4332317468824b6d885a62fb1a981aa79ae243a858b"}
Nov 04 00:10:32.968800 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:32.968764    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-114abfa61a32: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:32.968800 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:32.968795    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:32.969033 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:32.968863    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials podName:2f2072bb-1e8d-4eb3-9269-114981cc1d1e nodeName:}" failed. No retries permitted until 2026-11-04 00:10:36.968850216 +0000 UTC m=+2502.068450339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e") : secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:33.564733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.564708    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:33.680254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.680165    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69\" (UniqueName: \"kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69\") pod \"1dddeb40-2e71-456a-8fde-a9379b41a963\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") "
Nov 04 00:10:33.680254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.680230    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca\") pod \"1dddeb40-2e71-456a-8fde-a9379b41a963\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") "
Nov 04 00:10:33.680450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.680435    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca\") pod \"1dddeb40-2e71-456a-8fde-a9379b41a963\" (UID: \"1dddeb40-2e71-456a-8fde-a9379b41a963\") "
Nov 04 00:10:33.680812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.680783    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca" pod "1dddeb40-2e71-456a-8fde-a9379b41a963" (UID: "1dddeb40-2e71-456a-8fde-a9379b41a963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:33.681146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.681125    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:33.682464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.682439    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca" pod "1dddeb40-2e71-456a-8fde-a9379b41a963" (UID: "1dddeb40-2e71-456a-8fde-a9379b41a963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:10:33.682639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.682623    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69" pod "1dddeb40-2e71-456a-8fde-a9379b41a963" (UID: "1dddeb40-2e71-456a-8fde-a9379b41a963"). InnerVolumeSpecName "kube-api-access-2cr69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:33.782694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.782659    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cr69\" (UniqueName: \"kubernetes.io/projected/1dddeb40-2e71-456a-8fde-a9379b41a963-kube-api-access-2cr69\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:33.782694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:33.782690    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dddeb40-2e71-456a-8fde-a9379b41a963-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:34.365406 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.365344    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn" podUID="c652bf3c-a1da-4201-8b00-09166bb4ed73" containerName="payload-processing" containerID="cri-o://729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2" gracePeriod=25
Nov 04 00:10:34.405595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.405568    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"
Nov 04 00:10:34.405700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.405637    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf" event={"ID":"1dddeb40-2e71-456a-8fde-a9379b41a963","Type":"ContainerDied","Data":"a435a7b75bf94fe630b5b19e60910fbdf140ec6154d64ecdeb7b6dca460662f5"}
Nov 04 00:10:34.405700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.405681    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a435a7b75bf94fe630b5b19e60910fbdf140ec6154d64ecdeb7b6dca460662f5"
Nov 04 00:10:34.649714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.649685    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:10:34.798162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.798130    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials\") pod \"c652bf3c-a1da-4201-8b00-09166bb4ed73\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") "
Nov 04 00:10:34.798368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.798185    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume\") pod \"c652bf3c-a1da-4201-8b00-09166bb4ed73\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") "
Nov 04 00:10:34.798368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.798210    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume\") pod \"c652bf3c-a1da-4201-8b00-09166bb4ed73\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") "
Nov 04 00:10:34.798368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.798231    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h\") pod \"c652bf3c-a1da-4201-8b00-09166bb4ed73\" (UID: \"c652bf3c-a1da-4201-8b00-09166bb4ed73\") "
Nov 04 00:10:34.798660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.798631    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume" pod "c652bf3c-a1da-4201-8b00-09166bb4ed73" (UID: "c652bf3c-a1da-4201-8b00-09166bb4ed73"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:34.798796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.798663    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume" pod "c652bf3c-a1da-4201-8b00-09166bb4ed73" (UID: "c652bf3c-a1da-4201-8b00-09166bb4ed73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:34.799120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.799099    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:34.799173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.799122    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c652bf3c-a1da-4201-8b00-09166bb4ed73-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:34.800552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.800524    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials" pod "c652bf3c-a1da-4201-8b00-09166bb4ed73" (UID: "c652bf3c-a1da-4201-8b00-09166bb4ed73"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:34.800660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.800557    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h" pod "c652bf3c-a1da-4201-8b00-09166bb4ed73" (UID: "c652bf3c-a1da-4201-8b00-09166bb4ed73"). InnerVolumeSpecName "kube-api-access-pd69h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:34.901142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.901056    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:34.901142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:34.901090    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd69h\" (UniqueName: \"kubernetes.io/projected/c652bf3c-a1da-4201-8b00-09166bb4ed73-kube-api-access-pd69h\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:35.425256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.425216    5074 generic.go:356] "Generic (PLEG): container finished" podID="c652bf3c-a1da-4201-8b00-09166bb4ed73" containerID="729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2" exitCode=0
Nov 04 00:10:35.425728 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.425302    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"
Nov 04 00:10:35.426168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.425337    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn" event={"ID":"c652bf3c-a1da-4201-8b00-09166bb4ed73","Type":"ContainerDied","Data":"729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2"}
Nov 04 00:10:35.430327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.427275    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn" event={"ID":"c652bf3c-a1da-4201-8b00-09166bb4ed73","Type":"ContainerDied","Data":"0bf8c92fbbef998f17a0bdfe9bd20325f8c931036d124312479b3e5da2215aaf"}
Nov 04 00:10:35.430327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.427325    5074 scope.go:122] "RemoveContainer" containerID="729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2"
Nov 04 00:10:35.443318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.443287    5074 scope.go:122] "RemoveContainer" containerID="729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2"
Nov 04 00:10:35.443685 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:35.443660    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2\": container with ID starting with 729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2 not found: ID does not exist" containerID="729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2"
Nov 04 00:10:35.443799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.443695    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2"} err="failed to get container status \"729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2\": rpc error: code = NotFound desc = could not find container \"729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2\": container with ID starting with 729d271a72b82545454ab01b5d4e60e7966be9f32ad3a2104aa511329264eba2 not found: ID does not exist"
Nov 04 00:10:35.476548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.476515    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"]
Nov 04 00:10:35.487273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:35.487245    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7fb8sntqn"]
Nov 04 00:10:36.362074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.362037    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"]
Nov 04 00:10:36.362849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.362834    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c652bf3c-a1da-4201-8b00-09166bb4ed73" containerName="payload-processing"
Nov 04 00:10:36.362913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.362850    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c652bf3c-a1da-4201-8b00-09166bb4ed73" containerName="payload-processing"
Nov 04 00:10:36.362913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.362882    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1dddeb40-2e71-456a-8fde-a9379b41a963" containerName="revoke-keys"
Nov 04 00:10:36.362913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.362887    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dddeb40-2e71-456a-8fde-a9379b41a963" containerName="revoke-keys"
Nov 04 00:10:36.363012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.362986    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c652bf3c-a1da-4201-8b00-09166bb4ed73" containerName="payload-processing"
Nov 04 00:10:36.363012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.363000    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dddeb40-2e71-456a-8fde-a9379b41a963" containerName="revoke-keys"
Nov 04 00:10:36.373825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.373795    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.377122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.377092    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"]
Nov 04 00:10:36.388144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.388118    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:10:36.388260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.388120    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:10:36.420694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.420665    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjrn\" (UniqueName: \"kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.420890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.420809    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.420890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.420871    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.522275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.522236    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjrn\" (UniqueName: \"kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.522458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.522341    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.522458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.522407    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.523214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.523190    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.524780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.524758    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.535436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.535414    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjrn\" (UniqueName: \"kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn\") pod \"maas-api-revoke-sub-bb9eb6ee-fsvzx\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.694673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.694639    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:36.980578 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:36.980533    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf2c351_cb98_4ee5_942f_cc517b9a9511.slice/crio-99784ec8dbf727c0bd42b779fc50b5ea2c0cd8e64e630376e797b9a33a9ff413 WatchSource:0}: Error finding container 99784ec8dbf727c0bd42b779fc50b5ea2c0cd8e64e630376e797b9a33a9ff413: Status 404 returned error can't find the container with id 99784ec8dbf727c0bd42b779fc50b5ea2c0cd8e64e630376e797b9a33a9ff413
Nov 04 00:10:36.984498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:36.984476    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"]
Nov 04 00:10:37.019231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:37.019202    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c652bf3c-a1da-4201-8b00-09166bb4ed73" path="/var/lib/kubelet/pods/c652bf3c-a1da-4201-8b00-09166bb4ed73/volumes"
Nov 04 00:10:37.028431 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:37.028407    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-114abfa61a32: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:37.028528 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:37.028436    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:37.028528 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:37.028498    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials podName:2f2072bb-1e8d-4eb3-9269-114981cc1d1e nodeName:}" failed. No retries permitted until 2026-11-04 00:10:45.028484335 +0000 UTC m=+2510.128084481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e") : secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:37.446615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:37.446583    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx" event={"ID":"8bf2c351-cb98-4ee5-942f-cc517b9a9511","Type":"ContainerStarted","Data":"a7fc765d2ee6bb63e34d9bad59cdd4c20b0cd22d3753bd7a4510c2b71a956220"}
Nov 04 00:10:37.447123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:37.447095    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx" event={"ID":"8bf2c351-cb98-4ee5-942f-cc517b9a9511","Type":"ContainerStarted","Data":"99784ec8dbf727c0bd42b779fc50b5ea2c0cd8e64e630376e797b9a33a9ff413"}
Nov 04 00:10:37.517377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:37.517321    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx" podStartSLOduration=1.517308084 podStartE2EDuration="1.517308084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:37.480318152 +0000 UTC m=+2502.579918296" watchObservedRunningTime="2026-11-04 00:10:37.517308084 +0000 UTC m=+2502.616908228"
Nov 04 00:10:38.035971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:38.035936    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"]
Nov 04 00:10:38.040120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:38.040089    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23153751-gnw4m"]
Nov 04 00:10:38.458602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:38.458564    5074 generic.go:356] "Generic (PLEG): container finished" podID="8bf2c351-cb98-4ee5-942f-cc517b9a9511" containerID="a7fc765d2ee6bb63e34d9bad59cdd4c20b0cd22d3753bd7a4510c2b71a956220" exitCode=0
Nov 04 00:10:38.459055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:38.458653    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx" event={"ID":"8bf2c351-cb98-4ee5-942f-cc517b9a9511","Type":"ContainerDied","Data":"a7fc765d2ee6bb63e34d9bad59cdd4c20b0cd22d3753bd7a4510c2b71a956220"}
Nov 04 00:10:39.020969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.020933    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c7f9f1-299f-4b53-81d7-37ee15c7c5d3" path="/var/lib/kubelet/pods/75c7f9f1-299f-4b53-81d7-37ee15c7c5d3/volumes"
Nov 04 00:10:39.645334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.645301    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:39.765188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.765112    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca\") pod \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") "
Nov 04 00:10:39.765329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.765292    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca\") pod \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") "
Nov 04 00:10:39.765397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.765384    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn\" (UniqueName: \"kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn\") pod \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\" (UID: \"8bf2c351-cb98-4ee5-942f-cc517b9a9511\") "
Nov 04 00:10:39.765518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.765490    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca" pod "8bf2c351-cb98-4ee5-942f-cc517b9a9511" (UID: "8bf2c351-cb98-4ee5-942f-cc517b9a9511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:39.765809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.765791    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:39.767455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.767428    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca" pod "8bf2c351-cb98-4ee5-942f-cc517b9a9511" (UID: "8bf2c351-cb98-4ee5-942f-cc517b9a9511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:10:39.767542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.767528    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn" pod "8bf2c351-cb98-4ee5-942f-cc517b9a9511" (UID: "8bf2c351-cb98-4ee5-942f-cc517b9a9511"). InnerVolumeSpecName "kube-api-access-kpjrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:39.866904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.866874    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf2c351-cb98-4ee5-942f-cc517b9a9511-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:39.866904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:39.866901    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjrn\" (UniqueName: \"kubernetes.io/projected/8bf2c351-cb98-4ee5-942f-cc517b9a9511-kube-api-access-kpjrn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:40.477409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:40.477377    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"
Nov 04 00:10:40.477600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:40.477418    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx" event={"ID":"8bf2c351-cb98-4ee5-942f-cc517b9a9511","Type":"ContainerDied","Data":"99784ec8dbf727c0bd42b779fc50b5ea2c0cd8e64e630376e797b9a33a9ff413"}
Nov 04 00:10:40.477600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:40.477476    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99784ec8dbf727c0bd42b779fc50b5ea2c0cd8e64e630376e797b9a33a9ff413"
Nov 04 00:10:41.379898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:41.379869    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:41.451725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:41.451687    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"]
Nov 04 00:10:41.999862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:41.999824    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"]
Nov 04 00:10:42.633874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.633838    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"]
Nov 04 00:10:42.635620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.635586    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8bf2c351-cb98-4ee5-942f-cc517b9a9511" containerName="revoke-keys"
Nov 04 00:10:42.635816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.635801    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf2c351-cb98-4ee5-942f-cc517b9a9511" containerName="revoke-keys"
Nov 04 00:10:42.636160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.636146    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf2c351-cb98-4ee5-942f-cc517b9a9511" containerName="revoke-keys"
Nov 04 00:10:42.654769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.641073    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.676770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.676678    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.677092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.677061    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkg7\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.677365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.677341    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.677543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.677521    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.685345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.685315    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"]
Nov 04 00:10:42.689548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.689452    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-78dce14119dd\""
Nov 04 00:10:42.706472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.706440    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-e2e--9366bb2e-dockercfg-zdmjt\""
Nov 04 00:10:42.779274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.779242    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.779453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.779287    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.779501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.779450    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.779501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.779489    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkg7\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.780280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.780249    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.780411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.780287    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.781697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.781672    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:42.793339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:42.793308    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkg7\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:43.027926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.027890    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:43.391125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.391079    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"]
Nov 04 00:10:43.552812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.552772    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" event={"ID":"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4","Type":"ContainerStarted","Data":"9334ea8ced54fed9a6f1de46ade051dbaeb39db736cbdd88117204cc7f025166"}
Nov 04 00:10:43.553000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.552838    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" event={"ID":"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4","Type":"ContainerStarted","Data":"897ca2236140416cb555b7236f90a271e7add34d18baa897f6ae942cf1c7b888"}
Nov 04 00:10:43.553000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.552869    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:43.600895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.600841    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" podStartSLOduration=1.60082836 podStartE2EDuration="1.60082836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:43.59783479 +0000 UTC m=+2508.697434935" watchObservedRunningTime="2026-11-04 00:10:43.60082836 +0000 UTC m=+2508.700428503"
Nov 04 00:10:43.615089 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:43.615012    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-78dce14119dd: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:43.615270 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:43.615257    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:43.615424 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:43.615405    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:44.115386936 +0000 UTC m=+2509.214987061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:43.664405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:43.664369    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"]
Nov 04 00:10:44.130965 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:44.130922    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-78dce14119dd: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:44.130965 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:44.130958    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:44.131290 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:44.131019    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:45.131002125 +0000 UTC m=+2510.230602274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:44.561064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:44.561030    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" secret="" err="secret \"payload-processing-external-model-e2e--9366bb2e-dockercfg-zdmjt\" not found"
Nov 04 00:10:44.649365 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:44.645483    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-plugins-e2e-extmod-a-8895cb: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:44.649365 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:44.645570    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:45.145549041 +0000 UTC m=+2510.245149190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:45.051058 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.051021    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-114abfa61a32: secret "agc-cred-114abfa61a32" not found
Nov 04 00:10:45.051058 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.051053    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/e2e-cfl-140b39-provider-cred: secret "e2e-cfl-140b39-provider-cred" not found
Nov 04 00:10:45.051058 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.051063    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg: [secret "agc-cred-114abfa61a32" not found, secret "e2e-cfl-140b39-provider-cred" not found]
Nov 04 00:10:45.051539 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.051116    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials podName:2f2072bb-1e8d-4eb3-9269-114981cc1d1e nodeName:}" failed. No retries permitted until 2026-11-04 00:11:01.051101673 +0000 UTC m=+2526.150701796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e") : [secret "agc-cred-114abfa61a32" not found, secret "e2e-cfl-140b39-provider-cred" not found]
Nov 04 00:10:45.152628 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.152586    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-78dce14119dd: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:45.152628 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.152626    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:45.152934 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.152655    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-plugins-e2e-extmod-a-8895cb: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:45.152934 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.152687    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:47.152667378 +0000 UTC m=+2512.252267513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:45.152934 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:45.152730    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:46.152713309 +0000 UTC m=+2511.252313448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:46.036576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.036536    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"]
Nov 04 00:10:46.045049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.045015    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48229cab-lb7jq"]
Nov 04 00:10:46.166303 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:46.166270    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-plugins-e2e-extmod-a-8895cb: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:46.166701 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:46.166342    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:48.166328074 +0000 UTC m=+2513.265928223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:46.452452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.452408    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" podUID="2f2072bb-1e8d-4eb3-9269-114981cc1d1e" containerName="payload-processing" containerID="cri-o://545f0550c69deb52f7305864d5f0797740dbeae07a65d467636d06c26851b893" gracePeriod=25
Nov 04 00:10:46.580168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.580136    5074 generic.go:356] "Generic (PLEG): container finished" podID="2f2072bb-1e8d-4eb3-9269-114981cc1d1e" containerID="545f0550c69deb52f7305864d5f0797740dbeae07a65d467636d06c26851b893" exitCode=0
Nov 04 00:10:46.580329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.580211    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" event={"ID":"2f2072bb-1e8d-4eb3-9269-114981cc1d1e","Type":"ContainerDied","Data":"545f0550c69deb52f7305864d5f0797740dbeae07a65d467636d06c26851b893"}
Nov 04 00:10:46.759642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.759608    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:46.875627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.875591    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials\") pod \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") "
Nov 04 00:10:46.875828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.875722    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd\") pod \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") "
Nov 04 00:10:46.875828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.875765    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume\") pod \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") "
Nov 04 00:10:46.876008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.875976    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume\") pod \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\" (UID: \"2f2072bb-1e8d-4eb3-9269-114981cc1d1e\") "
Nov 04 00:10:46.876313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.876250    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume" pod "2f2072bb-1e8d-4eb3-9269-114981cc1d1e" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:46.876436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.876308    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume" pod "2f2072bb-1e8d-4eb3-9269-114981cc1d1e" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:46.876692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.876669    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:46.876794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.876700    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:46.877970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.877944    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials" pod "2f2072bb-1e8d-4eb3-9269-114981cc1d1e" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:46.878085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.878033    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd" pod "2f2072bb-1e8d-4eb3-9269-114981cc1d1e" (UID: "2f2072bb-1e8d-4eb3-9269-114981cc1d1e"). InnerVolumeSpecName "kube-api-access-tfvzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:46.978162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.978095    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:46.978162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:46.978123    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvzd\" (UniqueName: \"kubernetes.io/projected/2f2072bb-1e8d-4eb3-9269-114981cc1d1e-kube-api-access-tfvzd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:47.001344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.001293    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5" podUID="a858832b-4c68-4e1a-887a-49fe5bf22138" containerName="payload-processing" containerID="cri-o://b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84" gracePeriod=25
Nov 04 00:10:47.021413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.021387    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6636a4d9-8c65-49d3-a995-caf08906a24e" path="/var/lib/kubelet/pods/6636a4d9-8c65-49d3-a995-caf08906a24e/volumes"
Nov 04 00:10:47.209258 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:47.180636    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8895cb/agc-cred-78dce14119dd: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:47.209258 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:47.180666    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw: secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:47.209258 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:47.180734    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:51.180713665 +0000 UTC m=+2516.280313801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : secret "agc-cred-78dce14119dd" not found
Nov 04 00:10:47.268935 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.268907    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:47.382244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.382211    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume\") pod \"a858832b-4c68-4e1a-887a-49fe5bf22138\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") "
Nov 04 00:10:47.382412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.382356    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials\") pod \"a858832b-4c68-4e1a-887a-49fe5bf22138\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") "
Nov 04 00:10:47.382456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.382413    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc\") pod \"a858832b-4c68-4e1a-887a-49fe5bf22138\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") "
Nov 04 00:10:47.382520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.382505    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume\") pod \"a858832b-4c68-4e1a-887a-49fe5bf22138\" (UID: \"a858832b-4c68-4e1a-887a-49fe5bf22138\") "
Nov 04 00:10:47.382598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.382573    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume" pod "a858832b-4c68-4e1a-887a-49fe5bf22138" (UID: "a858832b-4c68-4e1a-887a-49fe5bf22138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:47.382901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.382880    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume" pod "a858832b-4c68-4e1a-887a-49fe5bf22138" (UID: "a858832b-4c68-4e1a-887a-49fe5bf22138"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:47.383101 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.383083    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:47.383161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.383103    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a858832b-4c68-4e1a-887a-49fe5bf22138-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:47.384625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.384598    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials" pod "a858832b-4c68-4e1a-887a-49fe5bf22138" (UID: "a858832b-4c68-4e1a-887a-49fe5bf22138"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:47.384774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.384636    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc" pod "a858832b-4c68-4e1a-887a-49fe5bf22138" (UID: "a858832b-4c68-4e1a-887a-49fe5bf22138"). InnerVolumeSpecName "kube-api-access-j5jhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:47.484594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.484561    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:47.484594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.484591    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jhc\" (UniqueName: \"kubernetes.io/projected/a858832b-4c68-4e1a-887a-49fe5bf22138-kube-api-access-j5jhc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:47.596702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.596614    5074 generic.go:356] "Generic (PLEG): container finished" podID="a858832b-4c68-4e1a-887a-49fe5bf22138" containerID="b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84" exitCode=0
Nov 04 00:10:47.596702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.596674    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"
Nov 04 00:10:47.596980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.596697    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5" event={"ID":"a858832b-4c68-4e1a-887a-49fe5bf22138","Type":"ContainerDied","Data":"b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84"}
Nov 04 00:10:47.596980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.596815    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5" event={"ID":"a858832b-4c68-4e1a-887a-49fe5bf22138","Type":"ContainerDied","Data":"53a5aab0f695f58cc56bc6af4ec70a7b7d1b0e7c18ac808dc0987e206c45f05f"}
Nov 04 00:10:47.596980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.596847    5074 scope.go:122] "RemoveContainer" containerID="b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84"
Nov 04 00:10:47.603717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.603691    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg" event={"ID":"2f2072bb-1e8d-4eb3-9269-114981cc1d1e","Type":"ContainerDied","Data":"d3bc4fd13bc45aa3c111ad9f9dae30e56bcaab5c4599f273c8d08da182018dab"}
Nov 04 00:10:47.603909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.603791    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"
Nov 04 00:10:47.614483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.614460    5074 scope.go:122] "RemoveContainer" containerID="b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84"
Nov 04 00:10:47.614868 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:47.614726    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84\": container with ID starting with b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84 not found: ID does not exist" containerID="b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84"
Nov 04 00:10:47.614942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.614882    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84"} err="failed to get container status \"b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84\": rpc error: code = NotFound desc = could not find container \"b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84\": container with ID starting with b961cdf8c93b9364e7f380b749359f6e33c6970016345e45baace65ddb50fd84 not found: ID does not exist"
Nov 04 00:10:47.614942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.614906    5074 scope.go:122] "RemoveContainer" containerID="545f0550c69deb52f7305864d5f0797740dbeae07a65d467636d06c26851b893"
Nov 04 00:10:47.806864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.806822    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"]
Nov 04 00:10:47.812711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.812685    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-c7498mmlg"]
Nov 04 00:10:47.825068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.825041    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"]
Nov 04 00:10:47.831507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.831483    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-6b96n2wr5"]
Nov 04 00:10:47.921786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.921695    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"]
Nov 04 00:10:47.922954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.922931    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a858832b-4c68-4e1a-887a-49fe5bf22138" containerName="payload-processing"
Nov 04 00:10:47.923017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.922957    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a858832b-4c68-4e1a-887a-49fe5bf22138" containerName="payload-processing"
Nov 04 00:10:47.923017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.923000    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2f2072bb-1e8d-4eb3-9269-114981cc1d1e" containerName="payload-processing"
Nov 04 00:10:47.923017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.923009    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2072bb-1e8d-4eb3-9269-114981cc1d1e" containerName="payload-processing"
Nov 04 00:10:47.923179 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.923167    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2072bb-1e8d-4eb3-9269-114981cc1d1e" containerName="payload-processing"
Nov 04 00:10:47.923215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.923195    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a858832b-4c68-4e1a-887a-49fe5bf22138" containerName="payload-processing"
Nov 04 00:10:47.929511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.929481    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:47.937291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:47.937260    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"]
Nov 04 00:10:48.095830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.095787    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.096032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.095889    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.096110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.096060    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2cqm\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.096305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.096278    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.197446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.197412    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.197622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.197529    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.197622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.197557    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.197622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.197617    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2cqm\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.197887 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:48.197866    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-plugins-e2e-extmod-a-8895cb: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:48.197994 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:48.197940    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume podName:6e2bbf41-a4b7-4c68-b898-16b8efbae4f4 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:52.197922456 +0000 UTC m=+2517.297522594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8895cb" not found
Nov 04 00:10:48.198359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.198335    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.198456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.198339    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.200261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.200230    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.211437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.211406    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2cqm\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm\") pod \"payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") " pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.258750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.258700    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:48.622946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.622889    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"]
Nov 04 00:10:48.938873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.938837    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"]
Nov 04 00:10:48.941054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:48.941022    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"]
Nov 04 00:10:49.021357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.021315    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2072bb-1e8d-4eb3-9269-114981cc1d1e" path="/var/lib/kubelet/pods/2f2072bb-1e8d-4eb3-9269-114981cc1d1e/volumes"
Nov 04 00:10:49.021921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.021902    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a858832b-4c68-4e1a-887a-49fe5bf22138" path="/var/lib/kubelet/pods/a858832b-4c68-4e1a-887a-49fe5bf22138/volumes"
Nov 04 00:10:49.040242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.040215    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"]
Nov 04 00:10:49.045438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.045413    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12179362-b6x7s"]
Nov 04 00:10:49.684528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.684486    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" event={"ID":"980ed745-edb5-41fd-8ac8-a9ffb70d20c3","Type":"ContainerStarted","Data":"ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914"}
Nov 04 00:10:49.685028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.684544    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" event={"ID":"980ed745-edb5-41fd-8ac8-a9ffb70d20c3","Type":"ContainerStarted","Data":"56bdcb9b7addbba18217583b09df345c5b11a26b8d728326e6ca8384e467b7f8"}
Nov 04 00:10:49.685028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.684567    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:49.685028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.684646    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" secret="" err="secret \"payload-processing-external-model-e2e--58feeaa8-dockercfg-sqpx4\" not found"
Nov 04 00:10:49.714926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:49.714880    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" podStartSLOduration=2.7148676 podStartE2EDuration="2.7148676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:10:49.713093289 +0000 UTC m=+2514.812693434" watchObservedRunningTime="2026-11-04 00:10:49.7148676 +0000 UTC m=+2514.814467741"
Nov 04 00:10:49.827821 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:49.827780    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-plugins-e2e-extmod-a-86deca: configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:49.827821 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:49.827807    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-86deca/agc-cred-836dc2249c00: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:49.827821 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:49.827822    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:49.828167 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:49.827875    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:50.32785635 +0000 UTC m=+2515.427456484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:49.828167 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:49.827899    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:50.327891237 +0000 UTC m=+2515.427491359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:50.335847 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:50.335811    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-plugins-e2e-extmod-a-86deca: configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:50.336029 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:50.335887    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:51.335870798 +0000 UTC m=+2516.435470927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:50.336029 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:50.335966    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-86deca/agc-cred-836dc2249c00: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:50.336029 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:50.335991    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:50.336173 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:50.336051    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:51.336035026 +0000 UTC m=+2516.435635148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:50.569792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.569754    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" podUID="6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" containerName="payload-processing" containerID="cri-o://9334ea8ced54fed9a6f1de46ade051dbaeb39db736cbdd88117204cc7f025166" gracePeriod=25
Nov 04 00:10:50.693958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.693930    5074 generic.go:356] "Generic (PLEG): container finished" podID="6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" containerID="9334ea8ced54fed9a6f1de46ade051dbaeb39db736cbdd88117204cc7f025166" exitCode=0
Nov 04 00:10:50.694292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.694008    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" event={"ID":"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4","Type":"ContainerDied","Data":"9334ea8ced54fed9a6f1de46ade051dbaeb39db736cbdd88117204cc7f025166"}
Nov 04 00:10:50.864943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.864879    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:50.944425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.944388    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials\") pod \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") "
Nov 04 00:10:50.944680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.944658    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7\") pod \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") "
Nov 04 00:10:50.944816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.944708    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume\") pod \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") "
Nov 04 00:10:50.944816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.944796    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume\") pod \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\" (UID: \"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4\") "
Nov 04 00:10:50.945194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.945167    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume" pod "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:50.945281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.945176    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume" pod "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:50.945408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.945392    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:50.945471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.945410    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:50.946642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.946616    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials" pod "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:50.947013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:50.946990    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7" pod "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" (UID: "6e2bbf41-a4b7-4c68-b898-16b8efbae4f4"). InnerVolumeSpecName "kube-api-access-cfkg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:51.020921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.020878    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dcc78a-5607-40cd-8db0-d71206b4ad7d" path="/var/lib/kubelet/pods/59dcc78a-5607-40cd-8db0-d71206b4ad7d/volumes"
Nov 04 00:10:51.047175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.047143    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkg7\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-kube-api-access-cfkg7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:51.047175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.047170    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:51.352140 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:51.352101    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-86deca/agc-cred-836dc2249c00: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:51.352140 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:51.352133    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:51.352373 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:51.352175    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-plugins-e2e-extmod-a-86deca: configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:51.352373 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:51.352191    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:53.352176008 +0000 UTC m=+2518.451776129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:51.352373 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:51.352231    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:53.352214961 +0000 UTC m=+2518.451815118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:51.705183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.705150    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw" event={"ID":"6e2bbf41-a4b7-4c68-b898-16b8efbae4f4","Type":"ContainerDied","Data":"897ca2236140416cb555b7236f90a271e7add34d18baa897f6ae942cf1c7b888"}
Nov 04 00:10:51.705183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.705176    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"
Nov 04 00:10:51.705777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.705211    5074 scope.go:122] "RemoveContainer" containerID="9334ea8ced54fed9a6f1de46ade051dbaeb39db736cbdd88117204cc7f025166"
Nov 04 00:10:51.750944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.750910    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"]
Nov 04 00:10:51.755800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:51.755775    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-566b69scw"]
Nov 04 00:10:53.018807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:53.018774    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" path="/var/lib/kubelet/pods/6e2bbf41-a4b7-4c68-b898-16b8efbae4f4/volumes"
Nov 04 00:10:53.380078 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:53.379987    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-plugins-e2e-extmod-a-86deca: configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:53.380078 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:53.380061    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:57.38004573 +0000 UTC m=+2522.479645856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-86deca" not found
Nov 04 00:10:53.380315 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:53.380102    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-86deca/agc-cred-836dc2249c00: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:53.380315 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:53.380124    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r: secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:53.380315 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:53.380192    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials podName:980ed745-edb5-41fd-8ac8-a9ffb70d20c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:10:57.380175993 +0000 UTC m=+2522.479776115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3") : secret "agc-cred-836dc2249c00" not found
Nov 04 00:10:53.941604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:53.941560    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9" podUID="5256dac2-e5d9-415d-b4bc-6b5860844a2a" containerName="payload-processing" containerID="cri-o://89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1" gracePeriod=25
Nov 04 00:10:54.246722 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.246691    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:10:54.408882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.408847    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9\") pod \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") "
Nov 04 00:10:54.409100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.408946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume\") pod \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") "
Nov 04 00:10:54.409100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.409007    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume\") pod \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") "
Nov 04 00:10:54.409239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.409219    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials\") pod \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\" (UID: \"5256dac2-e5d9-415d-b4bc-6b5860844a2a\") "
Nov 04 00:10:54.409474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.409434    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume" pod "5256dac2-e5d9-415d-b4bc-6b5860844a2a" (UID: "5256dac2-e5d9-415d-b4bc-6b5860844a2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:54.409474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.409450    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume" pod "5256dac2-e5d9-415d-b4bc-6b5860844a2a" (UID: "5256dac2-e5d9-415d-b4bc-6b5860844a2a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:54.409802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.409781    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:54.409802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.409800    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5256dac2-e5d9-415d-b4bc-6b5860844a2a-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:54.411274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.411252    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9" pod "5256dac2-e5d9-415d-b4bc-6b5860844a2a" (UID: "5256dac2-e5d9-415d-b4bc-6b5860844a2a"). InnerVolumeSpecName "kube-api-access-8l5w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:54.411412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.411387    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials" pod "5256dac2-e5d9-415d-b4bc-6b5860844a2a" (UID: "5256dac2-e5d9-415d-b4bc-6b5860844a2a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:54.510676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.510582    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5w9\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-kube-api-access-8l5w9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:54.510676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.510616    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5256dac2-e5d9-415d-b4bc-6b5860844a2a-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:54.734574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.734477    5074 generic.go:356] "Generic (PLEG): container finished" podID="5256dac2-e5d9-415d-b4bc-6b5860844a2a" containerID="89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1" exitCode=0
Nov 04 00:10:54.734821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.734592    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"
Nov 04 00:10:54.734821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.734593    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9" event={"ID":"5256dac2-e5d9-415d-b4bc-6b5860844a2a","Type":"ContainerDied","Data":"89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1"}
Nov 04 00:10:54.734821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.734718    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9" event={"ID":"5256dac2-e5d9-415d-b4bc-6b5860844a2a","Type":"ContainerDied","Data":"e5aa690de9fc8935b715a91fe9906d6915dbec46cc3290a3bdec21847ec946fa"}
Nov 04 00:10:54.734821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.734754    5074 scope.go:122] "RemoveContainer" containerID="89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1"
Nov 04 00:10:54.748642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.748616    5074 scope.go:122] "RemoveContainer" containerID="89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1"
Nov 04 00:10:54.748961 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:54.748938    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1\": container with ID starting with 89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1 not found: ID does not exist" containerID="89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1"
Nov 04 00:10:54.749036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.748968    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1"} err="failed to get container status \"89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1\": rpc error: code = NotFound desc = could not find container \"89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1\": container with ID starting with 89ce4842833061776aebfd76c272073d909c5f54dfea5fad00c685954d7ba1d1 not found: ID does not exist"
Nov 04 00:10:54.793951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.793900    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"]
Nov 04 00:10:54.803164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:54.803140    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6845qdvb9"]
Nov 04 00:10:55.020335 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:55.020292    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5256dac2-e5d9-415d-b4bc-6b5860844a2a" path="/var/lib/kubelet/pods/5256dac2-e5d9-415d-b4bc-6b5860844a2a/volumes"
Nov 04 00:10:55.695087 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:55.695046    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" podUID="980ed745-edb5-41fd-8ac8-a9ffb70d20c3" containerName="payload-processing" containerID="cri-o://ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914" gracePeriod=25
Nov 04 00:10:55.993312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:55.993279    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:56.046377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.046338    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume\") pod \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") "
Nov 04 00:10:56.046658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.046636    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm\") pod \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") "
Nov 04 00:10:56.046832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.046814    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume\") pod \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") "
Nov 04 00:10:56.047194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.047165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume" pod "980ed745-edb5-41fd-8ac8-a9ffb70d20c3" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:56.047296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.047256    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials\") pod \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\" (UID: \"980ed745-edb5-41fd-8ac8-a9ffb70d20c3\") "
Nov 04 00:10:56.047489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.047461    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume" pod "980ed745-edb5-41fd-8ac8-a9ffb70d20c3" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:56.048119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.048095    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:56.048238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.048225    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:56.050923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.050878    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"]
Nov 04 00:10:56.051799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.051775    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm" pod "980ed745-edb5-41fd-8ac8-a9ffb70d20c3" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3"). InnerVolumeSpecName "kube-api-access-h2cqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:56.052204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052180    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" containerName="payload-processing"
Nov 04 00:10:56.052294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052207    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" containerName="payload-processing"
Nov 04 00:10:56.052294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052280    5074 cpu_manager.go:414] "containerMap: removing container" podUID="980ed745-edb5-41fd-8ac8-a9ffb70d20c3" containerName="payload-processing"
Nov 04 00:10:56.052294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052290    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980ed745-edb5-41fd-8ac8-a9ffb70d20c3" containerName="payload-processing"
Nov 04 00:10:56.052453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052308    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5256dac2-e5d9-415d-b4bc-6b5860844a2a" containerName="payload-processing"
Nov 04 00:10:56.052453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052317    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5256dac2-e5d9-415d-b4bc-6b5860844a2a" containerName="payload-processing"
Nov 04 00:10:56.052558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052541    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2bbf41-a4b7-4c68-b898-16b8efbae4f4" containerName="payload-processing"
Nov 04 00:10:56.052611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052568    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="980ed745-edb5-41fd-8ac8-a9ffb70d20c3" containerName="payload-processing"
Nov 04 00:10:56.052611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.052579    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5256dac2-e5d9-415d-b4bc-6b5860844a2a" containerName="payload-processing"
Nov 04 00:10:56.056100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.056074    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials" pod "980ed745-edb5-41fd-8ac8-a9ffb70d20c3" (UID: "980ed745-edb5-41fd-8ac8-a9ffb70d20c3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:56.057187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.057165    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"]
Nov 04 00:10:56.057282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.057268    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.071995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.071969    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:10:56.072187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.072025    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:10:56.149254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.149211    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.149483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.149462    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvv5\" (UniqueName: \"kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.149617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.149598    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.149805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.149786    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2cqm\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-kube-api-access-h2cqm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:56.149865 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.149814    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/980ed745-edb5-41fd-8ac8-a9ffb70d20c3-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:56.250873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.250772    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.251031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.251016    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvv5\" (UniqueName: \"kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.251091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.251065    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.251493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.251468    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.253672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.253649    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.265354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.265331    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvv5\" (UniqueName: \"kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5\") pod \"maas-api-revoke-sub-7b46d674-7lmqv\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.401578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.401543    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:56.692908 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:10:56.692873    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649996dd_72e7_4914_9d22_b8c761f906b2.slice/crio-8c1dd065713b5ee7d531a419c033156da78c01c63d9d3c84a07d19d6f1bee35c WatchSource:0}: Error finding container 8c1dd065713b5ee7d531a419c033156da78c01c63d9d3c84a07d19d6f1bee35c: Status 404 returned error can't find the container with id 8c1dd065713b5ee7d531a419c033156da78c01c63d9d3c84a07d19d6f1bee35c
Nov 04 00:10:56.696364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.696340    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"]
Nov 04 00:10:56.757660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.757629    5074 generic.go:356] "Generic (PLEG): container finished" podID="980ed745-edb5-41fd-8ac8-a9ffb70d20c3" containerID="ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914" exitCode=0
Nov 04 00:10:56.757841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.757694    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"
Nov 04 00:10:56.757841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.757707    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" event={"ID":"980ed745-edb5-41fd-8ac8-a9ffb70d20c3","Type":"ContainerDied","Data":"ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914"}
Nov 04 00:10:56.757841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.757780    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r" event={"ID":"980ed745-edb5-41fd-8ac8-a9ffb70d20c3","Type":"ContainerDied","Data":"56bdcb9b7addbba18217583b09df345c5b11a26b8d728326e6ca8384e467b7f8"}
Nov 04 00:10:56.757841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.757796    5074 scope.go:122] "RemoveContainer" containerID="ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914"
Nov 04 00:10:56.759876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.759849    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv" event={"ID":"649996dd-72e7-4914-9d22-b8c761f906b2","Type":"ContainerStarted","Data":"8c1dd065713b5ee7d531a419c033156da78c01c63d9d3c84a07d19d6f1bee35c"}
Nov 04 00:10:56.774214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.774197    5074 scope.go:122] "RemoveContainer" containerID="ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914"
Nov 04 00:10:56.774495 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:56.774474    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914\": container with ID starting with ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914 not found: ID does not exist" containerID="ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914"
Nov 04 00:10:56.774587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.774501    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914"} err="failed to get container status \"ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914\": rpc error: code = NotFound desc = could not find container \"ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914\": container with ID starting with ed173cd7619a604eef8bab2788a22dbb01399c74217b960eb884f71999aa1914 not found: ID does not exist"
Nov 04 00:10:56.794655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.794623    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"]
Nov 04 00:10:56.802245 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:56.802217    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-86deca/payload-processing-external-model-e2e-extmod-a-86deca-6cf78bf4r"]
Nov 04 00:10:57.022310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:57.022228    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980ed745-edb5-41fd-8ac8-a9ffb70d20c3" path="/var/lib/kubelet/pods/980ed745-edb5-41fd-8ac8-a9ffb70d20c3/volumes"
Nov 04 00:10:57.771448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:57.771414    5074 generic.go:356] "Generic (PLEG): container finished" podID="649996dd-72e7-4914-9d22-b8c761f906b2" containerID="f0a5eb4bbc2727d748deea1c4dcc0758fb0286c7bcff45811587903491a56e02" exitCode=0
Nov 04 00:10:57.771866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:57.771501    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv" event={"ID":"649996dd-72e7-4914-9d22-b8c761f906b2","Type":"ContainerDied","Data":"f0a5eb4bbc2727d748deea1c4dcc0758fb0286c7bcff45811587903491a56e02"}
Nov 04 00:10:58.024109 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:10:58.024074    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:10:59.073169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.073093    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:59.215057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.215018    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5\" (UniqueName: \"kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5\") pod \"649996dd-72e7-4914-9d22-b8c761f906b2\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") "
Nov 04 00:10:59.215280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.215121    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca\") pod \"649996dd-72e7-4914-9d22-b8c761f906b2\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") "
Nov 04 00:10:59.215349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.215309    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca\") pod \"649996dd-72e7-4914-9d22-b8c761f906b2\" (UID: \"649996dd-72e7-4914-9d22-b8c761f906b2\") "
Nov 04 00:10:59.215576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.215533    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca" pod "649996dd-72e7-4914-9d22-b8c761f906b2" (UID: "649996dd-72e7-4914-9d22-b8c761f906b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:59.215916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.215889    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:59.217826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.217797    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5" pod "649996dd-72e7-4914-9d22-b8c761f906b2" (UID: "649996dd-72e7-4914-9d22-b8c761f906b2"). InnerVolumeSpecName "kube-api-access-rsvv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:59.218283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.218248    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca" pod "649996dd-72e7-4914-9d22-b8c761f906b2" (UID: "649996dd-72e7-4914-9d22-b8c761f906b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:10:59.333209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.319071    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvv5\" (UniqueName: \"kubernetes.io/projected/649996dd-72e7-4914-9d22-b8c761f906b2-kube-api-access-rsvv5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:59.333209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.319114    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649996dd-72e7-4914-9d22-b8c761f906b2-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:59.435320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.434792    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"]
Nov 04 00:10:59.436042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.436012    5074 cpu_manager.go:414] "containerMap: removing container" podUID="649996dd-72e7-4914-9d22-b8c761f906b2" containerName="revoke-keys"
Nov 04 00:10:59.436042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.436042    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649996dd-72e7-4914-9d22-b8c761f906b2" containerName="revoke-keys"
Nov 04 00:10:59.436272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.436255    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="649996dd-72e7-4914-9d22-b8c761f906b2" containerName="revoke-keys"
Nov 04 00:10:59.461532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.461495    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"]
Nov 04 00:10:59.461688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.461549    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.483344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.483310    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 04 00:10:59.483344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.483310    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 04 00:10:59.485902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.485879    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 04 00:10:59.485902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.485894    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 04 00:10:59.511290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.511257    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-97b64df7d06d\""
Nov 04 00:10:59.511290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.511289    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dcnfp\""
Nov 04 00:10:59.521906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.521871    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pjb\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.522068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.522043    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.522154 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.522134    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.522226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.522192    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.643048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.642820    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pjb\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.643307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.643273    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.643452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.643370    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.643522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.643450    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.644813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.644786    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.645822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.645798    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.675866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.671367    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.675866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.671462    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pjb\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb\") pod \"payload-processing-external-model-669765ccf8-n4j6q\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") " pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.783460 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.783431    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:10:59.791181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.791154    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"
Nov 04 00:10:59.791329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.791189    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv" event={"ID":"649996dd-72e7-4914-9d22-b8c761f906b2","Type":"ContainerDied","Data":"8c1dd065713b5ee7d531a419c033156da78c01c63d9d3c84a07d19d6f1bee35c"}
Nov 04 00:10:59.791329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:10:59.791278    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1dd065713b5ee7d531a419c033156da78c01c63d9d3c84a07d19d6f1bee35c"
Nov 04 00:11:00.076793 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:00.076761    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa463f53_c54d_4aac_b6cd_a78b6c2baaee.slice/crio-68c26cb6b5c8fada9e8353838f28cb5c48bbb4a1e7bb112892c973dbe453f141 WatchSource:0}: Error finding container 68c26cb6b5c8fada9e8353838f28cb5c48bbb4a1e7bb112892c973dbe453f141: Status 404 returned error can't find the container with id 68c26cb6b5c8fada9e8353838f28cb5c48bbb4a1e7bb112892c973dbe453f141
Nov 04 00:11:00.077643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:00.077621    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"]
Nov 04 00:11:00.802155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:00.802119    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q" event={"ID":"fa463f53-c54d-4aac-b6cd-a78b6c2baaee","Type":"ContainerStarted","Data":"3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825"}
Nov 04 00:11:00.802353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:00.802177    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:11:00.802353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:00.802198    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q" event={"ID":"fa463f53-c54d-4aac-b6cd-a78b6c2baaee","Type":"ContainerStarted","Data":"68c26cb6b5c8fada9e8353838f28cb5c48bbb4a1e7bb112892c973dbe453f141"}
Nov 04 00:11:00.836398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:00.836334    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q" podStartSLOduration=1.836319258 podStartE2EDuration="1.836319258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:00.831810518 +0000 UTC m=+2525.931410664" watchObservedRunningTime="2026-11-04 00:11:00.836319258 +0000 UTC m=+2525.935919401"
Nov 04 00:11:02.549905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.549864    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"]
Nov 04 00:11:02.567762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.565609    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.575773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.575209    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"]
Nov 04 00:11:02.692761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.690563    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.692761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.690658    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6f4\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.692761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.690716    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.692761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.690817    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.792489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.792452    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.792698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.792527    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6f4\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.792698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.792567    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.792698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.792640    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.793488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.793460    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.793556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.793545    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.795303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.795273    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.806626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.806564    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6f4\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:02.977449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:02.977415    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:03.281189 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:03.281153    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ebff03_f940_49fe_a278_bac9d75cabf7.slice/crio-cf5431c558f2ed517eb0b39549006040139922ac5272a8bdf5016024b31816b5 WatchSource:0}: Error finding container cf5431c558f2ed517eb0b39549006040139922ac5272a8bdf5016024b31816b5: Status 404 returned error can't find the container with id cf5431c558f2ed517eb0b39549006040139922ac5272a8bdf5016024b31816b5
Nov 04 00:11:03.283372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:03.283227    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"]
Nov 04 00:11:03.842506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:03.842464    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g" event={"ID":"99ebff03-f940-49fe-a278-bac9d75cabf7","Type":"ContainerStarted","Data":"fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f"}
Nov 04 00:11:03.843041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:03.842535    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g" event={"ID":"99ebff03-f940-49fe-a278-bac9d75cabf7","Type":"ContainerStarted","Data":"cf5431c558f2ed517eb0b39549006040139922ac5272a8bdf5016024b31816b5"}
Nov 04 00:11:03.843041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:03.842566    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:03.874509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:03.874452    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g" podStartSLOduration=1.874439183 podStartE2EDuration="1.874439183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:03.869779058 +0000 UTC m=+2528.969379199" watchObservedRunningTime="2026-11-04 00:11:03.874439183 +0000 UTC m=+2528.974039326"
Nov 04 00:11:05.974086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:05.974052    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"]
Nov 04 00:11:05.979375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:05.979354    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:05.988842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:05.988808    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"]
Nov 04 00:11:06.042773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.042706    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.042945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.042866    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.043021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.042989    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9nm\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.043094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.043075    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.047716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.047693    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f01fa1d21230\""
Nov 04 00:11:06.144096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.144056    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.144294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.144127    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.144294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.144234    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9nm\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.144428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.144293    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.144994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.144965    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.145140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.145114    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.146890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.146868    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.158303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.158274    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9nm\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm\") pod \"payload-processing-external-model-6749bc89cc-jhx5k\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") " pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.309720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.309632    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.596486 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:06.596453    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc09389_8b10_44ce_b7b2_25597720e473.slice/crio-f3a8a10817ad1fbe2a82dfc853d49539f3ae8c0dd1a074042522271978ad49b3 WatchSource:0}: Error finding container f3a8a10817ad1fbe2a82dfc853d49539f3ae8c0dd1a074042522271978ad49b3: Status 404 returned error can't find the container with id f3a8a10817ad1fbe2a82dfc853d49539f3ae8c0dd1a074042522271978ad49b3
Nov 04 00:11:06.601391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.601366    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"]
Nov 04 00:11:06.868708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.868629    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k" event={"ID":"2dc09389-8b10-44ce-b7b2-25597720e473","Type":"ContainerStarted","Data":"1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a"}
Nov 04 00:11:06.868708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.868685    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k" event={"ID":"2dc09389-8b10-44ce-b7b2-25597720e473","Type":"ContainerStarted","Data":"f3a8a10817ad1fbe2a82dfc853d49539f3ae8c0dd1a074042522271978ad49b3"}
Nov 04 00:11:06.868997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.868875    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:06.904514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:06.904467    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k" podStartSLOduration=1.904453483 podStartE2EDuration="1.904453483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:06.896122171 +0000 UTC m=+2531.995722315" watchObservedRunningTime="2026-11-04 00:11:06.904453483 +0000 UTC m=+2532.004053626"
Nov 04 00:11:07.804799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:07.787553    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"]
Nov 04 00:11:11.345525 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.345484    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"]
Nov 04 00:11:11.350311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.350291    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.359455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.359424    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"]
Nov 04 00:11:11.361074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.361025    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:11:11.361074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.361025    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:11:11.431144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.431110    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.431322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.431276    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.431402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.431387    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjld\" (UniqueName: \"kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.532688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.532649    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.532873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.532790    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.532920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.532882    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjld\" (UniqueName: \"kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.533527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.533492    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.535377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.535343    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.548611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.548587    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjld\" (UniqueName: \"kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld\") pod \"maas-api-revoke-sub-f7d32a3b-qbwng\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.671200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.671117    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:11.822149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.822122    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:11:11.967242 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:11.967194    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9314252_787b_4a9b_9b1b_7306f298f8bb.slice/crio-a1d281f694ca3bfae0f2e4224670fec585f24cbd04b45fabc7b391081cec7ccb WatchSource:0}: Error finding container a1d281f694ca3bfae0f2e4224670fec585f24cbd04b45fabc7b391081cec7ccb: Status 404 returned error can't find the container with id a1d281f694ca3bfae0f2e4224670fec585f24cbd04b45fabc7b391081cec7ccb
Nov 04 00:11:11.967675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:11.967648    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"]
Nov 04 00:11:12.788883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:12.788841    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw" podUID="467c9e4a-881a-4439-9633-bbd3047df4c6" containerName="payload-processing" containerID="cri-o://b117e3154faa03aeeb41ca74509ffbb355cc5e09d976b2d3dba52f444b298e8a" gracePeriod=25
Nov 04 00:11:12.931422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:12.931378    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng" event={"ID":"d9314252-787b-4a9b-9b1b-7306f298f8bb","Type":"ContainerStarted","Data":"76ec891129393e4bd6f4f2e291eab26d21d182cdd77bcb2a84457622f651e580"}
Nov 04 00:11:12.931553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:12.931469    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng" event={"ID":"d9314252-787b-4a9b-9b1b-7306f298f8bb","Type":"ContainerStarted","Data":"a1d281f694ca3bfae0f2e4224670fec585f24cbd04b45fabc7b391081cec7ccb"}
Nov 04 00:11:12.933248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:12.933221    5074 generic.go:356] "Generic (PLEG): container finished" podID="467c9e4a-881a-4439-9633-bbd3047df4c6" containerID="b117e3154faa03aeeb41ca74509ffbb355cc5e09d976b2d3dba52f444b298e8a" exitCode=0
Nov 04 00:11:12.933373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:12.933306    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw" event={"ID":"467c9e4a-881a-4439-9633-bbd3047df4c6","Type":"ContainerDied","Data":"b117e3154faa03aeeb41ca74509ffbb355cc5e09d976b2d3dba52f444b298e8a"}
Nov 04 00:11:13.045955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.045859    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"]
Nov 04 00:11:13.050075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.050041    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc99f5c-gnrlt"]
Nov 04 00:11:13.093754 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.093710    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:11:13.157587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.157549    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl\") pod \"467c9e4a-881a-4439-9633-bbd3047df4c6\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") "
Nov 04 00:11:13.157777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.157640    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume\") pod \"467c9e4a-881a-4439-9633-bbd3047df4c6\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") "
Nov 04 00:11:13.157777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.157668    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume\") pod \"467c9e4a-881a-4439-9633-bbd3047df4c6\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") "
Nov 04 00:11:13.157879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.157846    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials\") pod \"467c9e4a-881a-4439-9633-bbd3047df4c6\" (UID: \"467c9e4a-881a-4439-9633-bbd3047df4c6\") "
Nov 04 00:11:13.158108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.158078    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume" pod "467c9e4a-881a-4439-9633-bbd3047df4c6" (UID: "467c9e4a-881a-4439-9633-bbd3047df4c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:13.158234 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.158119    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume" pod "467c9e4a-881a-4439-9633-bbd3047df4c6" (UID: "467c9e4a-881a-4439-9633-bbd3047df4c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:13.158558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.158532    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:13.158648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.158565    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/467c9e4a-881a-4439-9633-bbd3047df4c6-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:13.160115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.160089    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl" pod "467c9e4a-881a-4439-9633-bbd3047df4c6" (UID: "467c9e4a-881a-4439-9633-bbd3047df4c6"). InnerVolumeSpecName "kube-api-access-k2hsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:13.160241 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.160145    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials" pod "467c9e4a-881a-4439-9633-bbd3047df4c6" (UID: "467c9e4a-881a-4439-9633-bbd3047df4c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:13.259600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.259555    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:13.259600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.259594    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hsl\" (UniqueName: \"kubernetes.io/projected/467c9e4a-881a-4439-9633-bbd3047df4c6-kube-api-access-k2hsl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:13.942526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.942489    5074 generic.go:356] "Generic (PLEG): container finished" podID="d9314252-787b-4a9b-9b1b-7306f298f8bb" containerID="76ec891129393e4bd6f4f2e291eab26d21d182cdd77bcb2a84457622f651e580" exitCode=0
Nov 04 00:11:13.943044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.942573    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng" event={"ID":"d9314252-787b-4a9b-9b1b-7306f298f8bb","Type":"ContainerDied","Data":"76ec891129393e4bd6f4f2e291eab26d21d182cdd77bcb2a84457622f651e580"}
Nov 04 00:11:13.944099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.944075    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw" event={"ID":"467c9e4a-881a-4439-9633-bbd3047df4c6","Type":"ContainerDied","Data":"92f801befe8f6d59446237f68c0a75a3ec3f98408ba2e304bdfd434668e58ce5"}
Nov 04 00:11:13.944222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.944095    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"
Nov 04 00:11:13.944222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.944124    5074 scope.go:122] "RemoveContainer" containerID="b117e3154faa03aeeb41ca74509ffbb355cc5e09d976b2d3dba52f444b298e8a"
Nov 04 00:11:13.979883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.979853    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"]
Nov 04 00:11:13.988123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:13.988097    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-7b792btlw"]
Nov 04 00:11:14.112494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.112468    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:14.272051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.271965    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca\") pod \"d9314252-787b-4a9b-9b1b-7306f298f8bb\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") "
Nov 04 00:11:14.272230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.272107    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca\") pod \"d9314252-787b-4a9b-9b1b-7306f298f8bb\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") "
Nov 04 00:11:14.272230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.272137    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld\" (UniqueName: \"kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld\") pod \"d9314252-787b-4a9b-9b1b-7306f298f8bb\" (UID: \"d9314252-787b-4a9b-9b1b-7306f298f8bb\") "
Nov 04 00:11:14.272550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.272521    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca" pod "d9314252-787b-4a9b-9b1b-7306f298f8bb" (UID: "d9314252-787b-4a9b-9b1b-7306f298f8bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:14.274241 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.274216    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld" pod "d9314252-787b-4a9b-9b1b-7306f298f8bb" (UID: "d9314252-787b-4a9b-9b1b-7306f298f8bb"). InnerVolumeSpecName "kube-api-access-rbjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:14.274325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.274223    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca" pod "d9314252-787b-4a9b-9b1b-7306f298f8bb" (UID: "d9314252-787b-4a9b-9b1b-7306f298f8bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:11:14.374177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.374138    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:14.374177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.374171    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjld\" (UniqueName: \"kubernetes.io/projected/d9314252-787b-4a9b-9b1b-7306f298f8bb-kube-api-access-rbjld\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:14.374177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.374183    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9314252-787b-4a9b-9b1b-7306f298f8bb-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:14.854310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.854279    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:14.925773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.925702    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"]
Nov 04 00:11:14.956424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.956390    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"
Nov 04 00:11:14.957020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.956424    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng" event={"ID":"d9314252-787b-4a9b-9b1b-7306f298f8bb","Type":"ContainerDied","Data":"a1d281f694ca3bfae0f2e4224670fec585f24cbd04b45fabc7b391081cec7ccb"}
Nov 04 00:11:14.957020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:14.956501    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d281f694ca3bfae0f2e4224670fec585f24cbd04b45fabc7b391081cec7ccb"
Nov 04 00:11:15.019867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:15.019832    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467c9e4a-881a-4439-9633-bbd3047df4c6" path="/var/lib/kubelet/pods/467c9e4a-881a-4439-9633-bbd3047df4c6/volumes"
Nov 04 00:11:15.020323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:15.020308    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58eae36-21c7-45a0-a772-38e347a81bdb" path="/var/lib/kubelet/pods/c58eae36-21c7-45a0-a772-38e347a81bdb/volumes"
Nov 04 00:11:17.881819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:17.881787    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:17.952513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:17.952478    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"]
Nov 04 00:11:19.926604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:19.926561    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c" podUID="1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" containerName="payload-processing" containerID="cri-o://622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167" gracePeriod=25
Nov 04 00:11:20.219952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.219918    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:11:20.260713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.260678    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials\") pod \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") "
Nov 04 00:11:20.260934 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.260833    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume\") pod \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") "
Nov 04 00:11:20.260934 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.260913    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk\") pod \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") "
Nov 04 00:11:20.261072 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.260966    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume\") pod \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\" (UID: \"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb\") "
Nov 04 00:11:20.261319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.261287    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume" pod "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" (UID: "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:20.261510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.261484    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:20.261636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.261545    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume" pod "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" (UID: "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:20.263333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.263309    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk" pod "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" (UID: "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb"). InnerVolumeSpecName "kube-api-access-c4ctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:20.263435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.263376    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials" pod "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" (UID: "1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:20.362333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.362297    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:20.362333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.362330    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ctk\" (UniqueName: \"kubernetes.io/projected/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-kube-api-access-c4ctk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:20.362333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:20.362341    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:21.018760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.018707    5074 generic.go:356] "Generic (PLEG): container finished" podID="1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" containerID="622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167" exitCode=0
Nov 04 00:11:21.019218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.018826    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"
Nov 04 00:11:21.021992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.021962    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c" event={"ID":"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb","Type":"ContainerDied","Data":"622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167"}
Nov 04 00:11:21.022136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.022022    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c" event={"ID":"1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb","Type":"ContainerDied","Data":"2e6d90b8787ae6fb4cba94fd0693ee30b6d0bccf6d01f894c30a6104fca74505"}
Nov 04 00:11:21.022136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.022053    5074 scope.go:122] "RemoveContainer" containerID="622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167"
Nov 04 00:11:21.040016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.039994    5074 scope.go:122] "RemoveContainer" containerID="622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167"
Nov 04 00:11:21.040353 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:21.040330    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167\": container with ID starting with 622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167 not found: ID does not exist" containerID="622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167"
Nov 04 00:11:21.040452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.040360    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167"} err="failed to get container status \"622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167\": rpc error: code = NotFound desc = could not find container \"622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167\": container with ID starting with 622db5bade28a2b7f5f10dd0ea940fdf322c2565e52c70611bfb61221c859167 not found: ID does not exist"
Nov 04 00:11:21.060823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.060793    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"]
Nov 04 00:11:21.082337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:21.082300    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7495xf58c"]
Nov 04 00:11:22.953637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:22.953589    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q" podUID="fa463f53-c54d-4aac-b6cd-a78b6c2baaee" containerName="payload-processing" containerID="cri-o://3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825" gracePeriod=25
Nov 04 00:11:23.022351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.022318    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" path="/var/lib/kubelet/pods/1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb/volumes"
Nov 04 00:11:23.231710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.231682    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:11:23.299989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.299960    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume\") pod \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") "
Nov 04 00:11:23.300174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.300095    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume\") pod \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") "
Nov 04 00:11:23.300174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.300132    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb\") pod \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") "
Nov 04 00:11:23.300374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.300344    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume" pod "fa463f53-c54d-4aac-b6cd-a78b6c2baaee" (UID: "fa463f53-c54d-4aac-b6cd-a78b6c2baaee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:23.300619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.300601    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:23.300795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.300754    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume" pod "fa463f53-c54d-4aac-b6cd-a78b6c2baaee" (UID: "fa463f53-c54d-4aac-b6cd-a78b6c2baaee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:23.302938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.302910    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb" pod "fa463f53-c54d-4aac-b6cd-a78b6c2baaee" (UID: "fa463f53-c54d-4aac-b6cd-a78b6c2baaee"). InnerVolumeSpecName "kube-api-access-54pjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:23.401197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.401162    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials\") pod \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\" (UID: \"fa463f53-c54d-4aac-b6cd-a78b6c2baaee\") "
Nov 04 00:11:23.401610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.401592    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:23.401706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.401613    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54pjb\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-kube-api-access-54pjb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:23.403291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.403265    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials" pod "fa463f53-c54d-4aac-b6cd-a78b6c2baaee" (UID: "fa463f53-c54d-4aac-b6cd-a78b6c2baaee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:23.502432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.502365    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa463f53-c54d-4aac-b6cd-a78b6c2baaee-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:23.943139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943099    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"]
Nov 04 00:11:23.943881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943864    5074 cpu_manager.go:414] "containerMap: removing container" podUID="467c9e4a-881a-4439-9633-bbd3047df4c6" containerName="payload-processing"
Nov 04 00:11:23.943881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943879    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467c9e4a-881a-4439-9633-bbd3047df4c6" containerName="payload-processing"
Nov 04 00:11:23.944062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943894    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" containerName="payload-processing"
Nov 04 00:11:23.944062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943900    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" containerName="payload-processing"
Nov 04 00:11:23.944062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943914    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa463f53-c54d-4aac-b6cd-a78b6c2baaee" containerName="payload-processing"
Nov 04 00:11:23.944062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943921    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa463f53-c54d-4aac-b6cd-a78b6c2baaee" containerName="payload-processing"
Nov 04 00:11:23.944062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943942    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d9314252-787b-4a9b-9b1b-7306f298f8bb" containerName="revoke-keys"
Nov 04 00:11:23.944062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.943947    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9314252-787b-4a9b-9b1b-7306f298f8bb" containerName="revoke-keys"
Nov 04 00:11:23.944417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.944074    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="467c9e4a-881a-4439-9633-bbd3047df4c6" containerName="payload-processing"
Nov 04 00:11:23.944417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.944086    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d232e3f-8f8c-4ac1-a3ca-2895bf5fdabb" containerName="payload-processing"
Nov 04 00:11:23.944417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.944094    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa463f53-c54d-4aac-b6cd-a78b6c2baaee" containerName="payload-processing"
Nov 04 00:11:23.944417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.944103    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9314252-787b-4a9b-9b1b-7306f298f8bb" containerName="revoke-keys"
Nov 04 00:11:23.950813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.949476    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:23.970341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.970254    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"]
Nov 04 00:11:23.975616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.975579    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:11:23.975813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:23.975598    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:11:24.009818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.008930    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcccn\" (UniqueName: \"kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.009818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.008984    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.009818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.009023    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.055238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.055203    5074 generic.go:356] "Generic (PLEG): container finished" podID="fa463f53-c54d-4aac-b6cd-a78b6c2baaee" containerID="3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825" exitCode=0
Nov 04 00:11:24.055439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.055272    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"
Nov 04 00:11:24.055439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.055378    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q" event={"ID":"fa463f53-c54d-4aac-b6cd-a78b6c2baaee","Type":"ContainerDied","Data":"3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825"}
Nov 04 00:11:24.055439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.055414    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q" event={"ID":"fa463f53-c54d-4aac-b6cd-a78b6c2baaee","Type":"ContainerDied","Data":"68c26cb6b5c8fada9e8353838f28cb5c48bbb4a1e7bb112892c973dbe453f141"}
Nov 04 00:11:24.055439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.055435    5074 scope.go:122] "RemoveContainer" containerID="3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825"
Nov 04 00:11:24.068327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.068290    5074 scope.go:122] "RemoveContainer" containerID="3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825"
Nov 04 00:11:24.068638 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:24.068609    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825\": container with ID starting with 3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825 not found: ID does not exist" containerID="3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825"
Nov 04 00:11:24.068695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.068650    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825"} err="failed to get container status \"3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825\": rpc error: code = NotFound desc = could not find container \"3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825\": container with ID starting with 3be8caa86e57a6ebcb2fc885eb84f73a69694631b92a8b853c3d977d66248825 not found: ID does not exist"
Nov 04 00:11:24.105397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.105366    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"]
Nov 04 00:11:24.111172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.111116    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcccn\" (UniqueName: \"kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.111373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.111175    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.111373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.111217    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.112475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.112444    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-669765ccf8-n4j6q"]
Nov 04 00:11:24.113397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.113371    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.117991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.117963    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.127796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.127770    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcccn\" (UniqueName: \"kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn\") pod \"maas-api-revoke-sub-cc6d9e99-wvb9z\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.300822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.299946    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:24.507995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.507605    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"]
Nov 04 00:11:24.515146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.515114    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.519914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.519841    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"]
Nov 04 00:11:24.523208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.523175    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfpk\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.523355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.523298    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.523355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.523330    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.523484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.523428    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.541771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.541225    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a43cca408b12\""
Nov 04 00:11:24.625302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.625210    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfpk\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.625462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.625329    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.625462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.625370    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.625462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.625457    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.626316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.626289    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.626467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.626293    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.628401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.628369    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.634693 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:24.634651    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9299a51c_bba7_424b_8f84_33acfd38ff31.slice/crio-158b64f44b3632a917570af67dc9dbf96695d81d07134d8c7c17d59938090be8 WatchSource:0}: Error finding container 158b64f44b3632a917570af67dc9dbf96695d81d07134d8c7c17d59938090be8: Status 404 returned error can't find the container with id 158b64f44b3632a917570af67dc9dbf96695d81d07134d8c7c17d59938090be8
Nov 04 00:11:24.639199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.639172    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"]
Nov 04 00:11:24.649788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.649761    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfpk\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk\") pod \"payload-processing-external-model-c4895777b-jwv7g\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") " pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:24.839208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:24.839175    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:25.022499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:25.022470    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa463f53-c54d-4aac-b6cd-a78b6c2baaee" path="/var/lib/kubelet/pods/fa463f53-c54d-4aac-b6cd-a78b6c2baaee/volumes"
Nov 04 00:11:25.066180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:25.066133    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z" event={"ID":"9299a51c-bba7-424b-8f84-33acfd38ff31","Type":"ContainerStarted","Data":"4db58e06a8ff835d7f2284bb39dac06f51080ceed61c679079f9cb923bd9377e"}
Nov 04 00:11:25.066371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:25.066209    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z" event={"ID":"9299a51c-bba7-424b-8f84-33acfd38ff31","Type":"ContainerStarted","Data":"158b64f44b3632a917570af67dc9dbf96695d81d07134d8c7c17d59938090be8"}
Nov 04 00:11:25.105131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:25.105065    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z" podStartSLOduration=2.105044532 podStartE2EDuration="2.105044532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:25.102399355 +0000 UTC m=+2550.201999500" watchObservedRunningTime="2026-11-04 00:11:25.105044532 +0000 UTC m=+2550.204644677"
Nov 04 00:11:25.133366 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:25.132293    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846b7c16_9dcc_493e_ba2a_0ee844093a4f.slice/crio-41d036137aa4724a24a558afd9ecb5540beedbc3c23829f3ee06e7196a2c1554 WatchSource:0}: Error finding container 41d036137aa4724a24a558afd9ecb5540beedbc3c23829f3ee06e7196a2c1554: Status 404 returned error can't find the container with id 41d036137aa4724a24a558afd9ecb5540beedbc3c23829f3ee06e7196a2c1554
Nov 04 00:11:25.138314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:25.138289    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"]
Nov 04 00:11:26.075390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:26.075347    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g" event={"ID":"846b7c16-9dcc-493e-ba2a-0ee844093a4f","Type":"ContainerStarted","Data":"4572a3e8c1b1d50f37334ad8e7301c00214c1afb434af196d8ed70f7514926bc"}
Nov 04 00:11:26.075892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:26.075437    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g" event={"ID":"846b7c16-9dcc-493e-ba2a-0ee844093a4f","Type":"ContainerStarted","Data":"41d036137aa4724a24a558afd9ecb5540beedbc3c23829f3ee06e7196a2c1554"}
Nov 04 00:11:26.075892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:26.075463    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:26.077196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:26.077165    5074 generic.go:356] "Generic (PLEG): container finished" podID="9299a51c-bba7-424b-8f84-33acfd38ff31" containerID="4db58e06a8ff835d7f2284bb39dac06f51080ceed61c679079f9cb923bd9377e" exitCode=0
Nov 04 00:11:26.077332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:26.077222    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z" event={"ID":"9299a51c-bba7-424b-8f84-33acfd38ff31","Type":"ContainerDied","Data":"4db58e06a8ff835d7f2284bb39dac06f51080ceed61c679079f9cb923bd9377e"}
Nov 04 00:11:26.120388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:26.120329    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g" podStartSLOduration=2.120314717 podStartE2EDuration="2.120314717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:26.117900978 +0000 UTC m=+2551.217501124" watchObservedRunningTime="2026-11-04 00:11:26.120314717 +0000 UTC m=+2551.219914861"
Nov 04 00:11:27.264634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.264600    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:27.275214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.275177    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn\" (UniqueName: \"kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn\") pod \"9299a51c-bba7-424b-8f84-33acfd38ff31\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") "
Nov 04 00:11:27.275406 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.275269    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca\") pod \"9299a51c-bba7-424b-8f84-33acfd38ff31\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") "
Nov 04 00:11:27.275406 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.275329    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca\") pod \"9299a51c-bba7-424b-8f84-33acfd38ff31\" (UID: \"9299a51c-bba7-424b-8f84-33acfd38ff31\") "
Nov 04 00:11:27.275824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.275791    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca" pod "9299a51c-bba7-424b-8f84-33acfd38ff31" (UID: "9299a51c-bba7-424b-8f84-33acfd38ff31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:27.276009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.275987    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:27.277693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.277665    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn" pod "9299a51c-bba7-424b-8f84-33acfd38ff31" (UID: "9299a51c-bba7-424b-8f84-33acfd38ff31"). InnerVolumeSpecName "kube-api-access-pcccn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:27.277815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.277706    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca" pod "9299a51c-bba7-424b-8f84-33acfd38ff31" (UID: "9299a51c-bba7-424b-8f84-33acfd38ff31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:11:27.377426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.377327    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcccn\" (UniqueName: \"kubernetes.io/projected/9299a51c-bba7-424b-8f84-33acfd38ff31-kube-api-access-pcccn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:27.377426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:27.377362    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9299a51c-bba7-424b-8f84-33acfd38ff31-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:28.024143 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:28.024098    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:11:28.094695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:28.094668    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"
Nov 04 00:11:28.094871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:28.094668    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z" event={"ID":"9299a51c-bba7-424b-8f84-33acfd38ff31","Type":"ContainerDied","Data":"158b64f44b3632a917570af67dc9dbf96695d81d07134d8c7c17d59938090be8"}
Nov 04 00:11:28.094871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:28.094811    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="158b64f44b3632a917570af67dc9dbf96695d81d07134d8c7c17d59938090be8"
Nov 04 00:11:35.812117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:35.812033    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"]
Nov 04 00:11:37.089273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:37.089242    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:11:37.175201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:37.175161    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"]
Nov 04 00:11:40.813022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.812982    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g" podUID="99ebff03-f940-49fe-a278-bac9d75cabf7" containerName="payload-processing" containerID="cri-o://fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f" gracePeriod=25
Nov 04 00:11:40.949605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.949562    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"]
Nov 04 00:11:40.953681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.951109    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9299a51c-bba7-424b-8f84-33acfd38ff31" containerName="revoke-keys"
Nov 04 00:11:40.953681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.951138    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9299a51c-bba7-424b-8f84-33acfd38ff31" containerName="revoke-keys"
Nov 04 00:11:40.953681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.951317    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9299a51c-bba7-424b-8f84-33acfd38ff31" containerName="revoke-keys"
Nov 04 00:11:40.958885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.958858    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:40.982146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.982093    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:11:40.985080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.985040    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:11:40.985414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:40.985390    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"]
Nov 04 00:11:41.099528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.099430    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqqrq\" (UniqueName: \"kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.099733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.099546    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.099877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.099830    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.143870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.143841    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:41.201362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.201309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.201580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.201396    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqqrq\" (UniqueName: \"kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.201580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.201454    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.202275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.202243    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.204044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.204023    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.215404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.215384    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqqrq\" (UniqueName: \"kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq\") pod \"maas-api-revoke-sub-2fad6aad-dbg6g\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.226556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.226527    5074 generic.go:356] "Generic (PLEG): container finished" podID="99ebff03-f940-49fe-a278-bac9d75cabf7" containerID="fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f" exitCode=0
Nov 04 00:11:41.226652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.226569    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"
Nov 04 00:11:41.226652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.226593    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g" event={"ID":"99ebff03-f940-49fe-a278-bac9d75cabf7","Type":"ContainerDied","Data":"fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f"}
Nov 04 00:11:41.226652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.226642    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g" event={"ID":"99ebff03-f940-49fe-a278-bac9d75cabf7","Type":"ContainerDied","Data":"cf5431c558f2ed517eb0b39549006040139922ac5272a8bdf5016024b31816b5"}
Nov 04 00:11:41.226799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.226675    5074 scope.go:122] "RemoveContainer" containerID="fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f"
Nov 04 00:11:41.243053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.243029    5074 scope.go:122] "RemoveContainer" containerID="fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f"
Nov 04 00:11:41.243382 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:41.243356    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f\": container with ID starting with fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f not found: ID does not exist" containerID="fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f"
Nov 04 00:11:41.243555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.243389    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f"} err="failed to get container status \"fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f\": rpc error: code = NotFound desc = could not find container \"fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f\": container with ID starting with fa3af21de0063386e52b458e53261373819ef55c4cc2a0b710750e193591579f not found: ID does not exist"
Nov 04 00:11:41.294665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.294632    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:41.302733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.302703    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume\") pod \"99ebff03-f940-49fe-a278-bac9d75cabf7\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") "
Nov 04 00:11:41.302873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.302852    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials\") pod \"99ebff03-f940-49fe-a278-bac9d75cabf7\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") "
Nov 04 00:11:41.302931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.302915    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume\") pod \"99ebff03-f940-49fe-a278-bac9d75cabf7\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") "
Nov 04 00:11:41.303093 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.303076    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4\") pod \"99ebff03-f940-49fe-a278-bac9d75cabf7\" (UID: \"99ebff03-f940-49fe-a278-bac9d75cabf7\") "
Nov 04 00:11:41.303290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.303262    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume" pod "99ebff03-f940-49fe-a278-bac9d75cabf7" (UID: "99ebff03-f940-49fe-a278-bac9d75cabf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:41.303447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.303423    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume" pod "99ebff03-f940-49fe-a278-bac9d75cabf7" (UID: "99ebff03-f940-49fe-a278-bac9d75cabf7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:41.303589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.303572    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:41.303651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.303592    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ebff03-f940-49fe-a278-bac9d75cabf7-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:41.305079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.305056    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials" pod "99ebff03-f940-49fe-a278-bac9d75cabf7" (UID: "99ebff03-f940-49fe-a278-bac9d75cabf7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:41.305182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.305159    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4" pod "99ebff03-f940-49fe-a278-bac9d75cabf7" (UID: "99ebff03-f940-49fe-a278-bac9d75cabf7"). InnerVolumeSpecName "kube-api-access-8t6f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:41.405778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.405730    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6f4\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-kube-api-access-8t6f4\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:41.405778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.405776    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ebff03-f940-49fe-a278-bac9d75cabf7-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:41.563103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.563069    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"]
Nov 04 00:11:41.568637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.568612    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-bd8fk9g4g"]
Nov 04 00:11:41.583469 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:41.583438    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa686443_8666_4e78_a8d5_66e851c7783a.slice/crio-2a8ee85b5964fd64be7b6b13883a5aca642618e1093eecafbe712bbf7678db9e WatchSource:0}: Error finding container 2a8ee85b5964fd64be7b6b13883a5aca642618e1093eecafbe712bbf7678db9e: Status 404 returned error can't find the container with id 2a8ee85b5964fd64be7b6b13883a5aca642618e1093eecafbe712bbf7678db9e
Nov 04 00:11:41.586761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:41.586716    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"]
Nov 04 00:11:42.176494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.176446    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k" podUID="2dc09389-8b10-44ce-b7b2-25597720e473" containerName="payload-processing" containerID="cri-o://1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a" gracePeriod=25
Nov 04 00:11:42.238383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.238350    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g" event={"ID":"fa686443-8666-4e78-a8d5-66e851c7783a","Type":"ContainerStarted","Data":"d889b965ee2e7fc648adcc77f3fbe0d1289e2aa1c5d35da2b80ea3a41ff63fbb"}
Nov 04 00:11:42.238513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.238404    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g" event={"ID":"fa686443-8666-4e78-a8d5-66e851c7783a","Type":"ContainerStarted","Data":"2a8ee85b5964fd64be7b6b13883a5aca642618e1093eecafbe712bbf7678db9e"}
Nov 04 00:11:42.264083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.263991    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g" podStartSLOduration=2.263979456 podStartE2EDuration="2.263979456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:42.262994032 +0000 UTC m=+2567.362594177" watchObservedRunningTime="2026-11-04 00:11:42.263979456 +0000 UTC m=+2567.363579599"
Nov 04 00:11:42.475485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.475450    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:42.520759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.520696    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials\") pod \"2dc09389-8b10-44ce-b7b2-25597720e473\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") "
Nov 04 00:11:42.520915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.520818    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm\") pod \"2dc09389-8b10-44ce-b7b2-25597720e473\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") "
Nov 04 00:11:42.520915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.520869    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume\") pod \"2dc09389-8b10-44ce-b7b2-25597720e473\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") "
Nov 04 00:11:42.521090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.521070    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume\") pod \"2dc09389-8b10-44ce-b7b2-25597720e473\" (UID: \"2dc09389-8b10-44ce-b7b2-25597720e473\") "
Nov 04 00:11:42.521962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.521935    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume" pod "2dc09389-8b10-44ce-b7b2-25597720e473" (UID: "2dc09389-8b10-44ce-b7b2-25597720e473"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:42.523089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.522796    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume" pod "2dc09389-8b10-44ce-b7b2-25597720e473" (UID: "2dc09389-8b10-44ce-b7b2-25597720e473"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:42.526286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.526254    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials" pod "2dc09389-8b10-44ce-b7b2-25597720e473" (UID: "2dc09389-8b10-44ce-b7b2-25597720e473"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:42.530109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.530078    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm" pod "2dc09389-8b10-44ce-b7b2-25597720e473" (UID: "2dc09389-8b10-44ce-b7b2-25597720e473"). InnerVolumeSpecName "kube-api-access-xv9nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:42.622733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.622700    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv9nm\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-kube-api-access-xv9nm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:42.622733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.622732    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:42.622733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.622757    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2dc09389-8b10-44ce-b7b2-25597720e473-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:42.623013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:42.622771    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2dc09389-8b10-44ce-b7b2-25597720e473-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:43.019719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.019674    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ebff03-f940-49fe-a278-bac9d75cabf7" path="/var/lib/kubelet/pods/99ebff03-f940-49fe-a278-bac9d75cabf7/volumes"
Nov 04 00:11:43.248520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.248477    5074 generic.go:356] "Generic (PLEG): container finished" podID="2dc09389-8b10-44ce-b7b2-25597720e473" containerID="1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a" exitCode=0
Nov 04 00:11:43.248962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.248544    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"
Nov 04 00:11:43.248962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.248564    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k" event={"ID":"2dc09389-8b10-44ce-b7b2-25597720e473","Type":"ContainerDied","Data":"1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a"}
Nov 04 00:11:43.248962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.248617    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k" event={"ID":"2dc09389-8b10-44ce-b7b2-25597720e473","Type":"ContainerDied","Data":"f3a8a10817ad1fbe2a82dfc853d49539f3ae8c0dd1a074042522271978ad49b3"}
Nov 04 00:11:43.248962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.248637    5074 scope.go:122] "RemoveContainer" containerID="1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a"
Nov 04 00:11:43.250977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.250948    5074 generic.go:356] "Generic (PLEG): container finished" podID="fa686443-8666-4e78-a8d5-66e851c7783a" containerID="d889b965ee2e7fc648adcc77f3fbe0d1289e2aa1c5d35da2b80ea3a41ff63fbb" exitCode=0
Nov 04 00:11:43.251146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.251121    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g" event={"ID":"fa686443-8666-4e78-a8d5-66e851c7783a","Type":"ContainerDied","Data":"d889b965ee2e7fc648adcc77f3fbe0d1289e2aa1c5d35da2b80ea3a41ff63fbb"}
Nov 04 00:11:43.262673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.262653    5074 scope.go:122] "RemoveContainer" containerID="1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a"
Nov 04 00:11:43.262991 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:43.262970    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a\": container with ID starting with 1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a not found: ID does not exist" containerID="1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a"
Nov 04 00:11:43.263078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.263001    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a"} err="failed to get container status \"1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a\": rpc error: code = NotFound desc = could not find container \"1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a\": container with ID starting with 1181dfd4cb96bf9383c65e6a72d21493b46fcbf69005d1124fa8a6d84032f63a not found: ID does not exist"
Nov 04 00:11:43.316780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.316690    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"]
Nov 04 00:11:43.356080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:43.356044    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6749bc89cc-jhx5k"]
Nov 04 00:11:44.433615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.433579    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:44.446833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.446804    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq\" (UniqueName: \"kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq\") pod \"fa686443-8666-4e78-a8d5-66e851c7783a\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") "
Nov 04 00:11:44.446995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.446849    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca\") pod \"fa686443-8666-4e78-a8d5-66e851c7783a\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") "
Nov 04 00:11:44.447046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.447000    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca\") pod \"fa686443-8666-4e78-a8d5-66e851c7783a\" (UID: \"fa686443-8666-4e78-a8d5-66e851c7783a\") "
Nov 04 00:11:44.447423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.447385    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca" pod "fa686443-8666-4e78-a8d5-66e851c7783a" (UID: "fa686443-8666-4e78-a8d5-66e851c7783a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:44.447811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.447785    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:44.449199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.449169    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq" pod "fa686443-8666-4e78-a8d5-66e851c7783a" (UID: "fa686443-8666-4e78-a8d5-66e851c7783a"). InnerVolumeSpecName "kube-api-access-cqqrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:44.449409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.449386    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca" pod "fa686443-8666-4e78-a8d5-66e851c7783a" (UID: "fa686443-8666-4e78-a8d5-66e851c7783a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:11:44.549605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.549521    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqqrq\" (UniqueName: \"kubernetes.io/projected/fa686443-8666-4e78-a8d5-66e851c7783a-kube-api-access-cqqrq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:44.549605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:44.549556    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa686443-8666-4e78-a8d5-66e851c7783a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:45.020506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:45.020469    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc09389-8b10-44ce-b7b2-25597720e473" path="/var/lib/kubelet/pods/2dc09389-8b10-44ce-b7b2-25597720e473/volumes"
Nov 04 00:11:45.271020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:45.270929    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g" event={"ID":"fa686443-8666-4e78-a8d5-66e851c7783a","Type":"ContainerDied","Data":"2a8ee85b5964fd64be7b6b13883a5aca642618e1093eecafbe712bbf7678db9e"}
Nov 04 00:11:45.271020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:45.270988    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8ee85b5964fd64be7b6b13883a5aca642618e1093eecafbe712bbf7678db9e"
Nov 04 00:11:45.271020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:45.270945    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"
Nov 04 00:11:51.566376 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.566341    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"]
Nov 04 00:11:51.567129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567110    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2dc09389-8b10-44ce-b7b2-25597720e473" containerName="payload-processing"
Nov 04 00:11:51.567129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567128    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc09389-8b10-44ce-b7b2-25597720e473" containerName="payload-processing"
Nov 04 00:11:51.567252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567150    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa686443-8666-4e78-a8d5-66e851c7783a" containerName="revoke-keys"
Nov 04 00:11:51.567252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567156    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa686443-8666-4e78-a8d5-66e851c7783a" containerName="revoke-keys"
Nov 04 00:11:51.567252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567183    5074 cpu_manager.go:414] "containerMap: removing container" podUID="99ebff03-f940-49fe-a278-bac9d75cabf7" containerName="payload-processing"
Nov 04 00:11:51.567252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567188    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ebff03-f940-49fe-a278-bac9d75cabf7" containerName="payload-processing"
Nov 04 00:11:51.567418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567300    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ebff03-f940-49fe-a278-bac9d75cabf7" containerName="payload-processing"
Nov 04 00:11:51.567418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567312    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc09389-8b10-44ce-b7b2-25597720e473" containerName="payload-processing"
Nov 04 00:11:51.567418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.567322    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa686443-8666-4e78-a8d5-66e851c7783a" containerName="revoke-keys"
Nov 04 00:11:51.571267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.571247    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.604915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.604879    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"]
Nov 04 00:11:51.655176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655135    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655193    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6lt\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655307    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655413    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655441    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655473    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655564    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655628    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.655709 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.655677    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.757611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.757562    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.757829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.757691    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.757829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.757802    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.757990    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758201    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6lt\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758350    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758515    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758392    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758727    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.758890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758867    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.759068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758612    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.759166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.758882    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.759509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.759489    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.759614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.759535    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.768182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.768154    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.769587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.769286    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.772970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.772946    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.775820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.775796    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6lt\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ln7lf\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:51.890885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:51.890799    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:52.005784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.001922    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"]
Nov 04 00:11:52.022772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.022077    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.039920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.032941    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9dde98e016dd\""
Nov 04 00:11:52.039920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.039191    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"]
Nov 04 00:11:52.161896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.151039    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"]
Nov 04 00:11:52.161896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.158634    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.187285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.186730    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:11:52.191249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.189359    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:11:52.219793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.200610    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8sls\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.219793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.217853    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.219793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.217985    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.219793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.209651    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"]
Nov 04 00:11:52.219793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.219562    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.256553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.256509    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"]
Nov 04 00:11:52.270143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.265964    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.283989 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:52.282235    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324fe1c5_df49_4e36_92d0_f4dccead361a.slice/crio-3b385eb75bab47960952c254e43a24447d68392580a7869b867a0c6306ad55a7 WatchSource:0}: Error finding container 3b385eb75bab47960952c254e43a24447d68392580a7869b867a0c6306ad55a7: Status 404 returned error can't find the container with id 3b385eb75bab47960952c254e43a24447d68392580a7869b867a0c6306ad55a7
Nov 04 00:11:52.289472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.288227    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:11:52.289472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.288315    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:11:52.289472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.288361    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:11:52.298663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.298526    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"]
Nov 04 00:11:52.309647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.309613    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"]
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.321482    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.321544    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.321640    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8sls\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.321703    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.322031    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.322109    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brr5z\" (UniqueName: \"kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.322201    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.323912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.323140    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.325311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.325275    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.333469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.333439    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.337918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.337894    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8sls\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls\") pod \"payload-processing-external-model-78d75b4487-g7gmr\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") " pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.345801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.345774    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" event={"ID":"324fe1c5-df49-4e36-92d0-f4dccead361a","Type":"ContainerStarted","Data":"3b385eb75bab47960952c254e43a24447d68392580a7869b867a0c6306ad55a7"}
Nov 04 00:11:52.356469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.356436    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:52.423700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.423664    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brr5z\" (UniqueName: \"kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.423863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.423809    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.423938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.423865    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.423938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.423917    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.424054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.423975    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7jc\" (UniqueName: \"kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.424111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.424074    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.425123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.425098    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.428381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.427981    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.451274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.448943    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brr5z\" (UniqueName: \"kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z\") pod \"maas-api-revoke-sub-e0b3eaae-hv7rs\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.518775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.514586    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:52.526546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.525629    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7jc\" (UniqueName: \"kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.526546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.525727    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.526546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.525921    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.528099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.528075    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.532224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.532177    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.547799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.545434    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7jc\" (UniqueName: \"kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc\") pod \"maas-api-revoke-sub-95b8e35a-9xwdb\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.614897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.612770    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:52.684711 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:52.684518    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbe1cbe_9710_4d1a_9d60_1599c65446f6.slice/crio-724443d5d218c4336d88fa4ba2e1e8d934827bd528705691f6d38c68f486a074 WatchSource:0}: Error finding container 724443d5d218c4336d88fa4ba2e1e8d934827bd528705691f6d38c68f486a074: Status 404 returned error can't find the container with id 724443d5d218c4336d88fa4ba2e1e8d934827bd528705691f6d38c68f486a074
Nov 04 00:11:52.704301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.704267    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"]
Nov 04 00:11:52.880150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.880117    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"]
Nov 04 00:11:52.949897 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:11:52.949870    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a507c6_2a8a_4e46_b3ec_1dd392193c04.slice/crio-31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59 WatchSource:0}: Error finding container 31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59: Status 404 returned error can't find the container with id 31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59
Nov 04 00:11:52.953511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:52.953487    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"]
Nov 04 00:11:53.360890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.360848    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs" event={"ID":"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c","Type":"ContainerStarted","Data":"37ba15853158318a0d11d6aefbec0476a32d0a322adfa9936aac076f76122360"}
Nov 04 00:11:53.362699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.362661    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb" event={"ID":"83a507c6-2a8a-4e46-b3ec-1dd392193c04","Type":"ContainerStarted","Data":"31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59"}
Nov 04 00:11:53.364678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.364647    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr" event={"ID":"2fbe1cbe-9710-4d1a-9d60-1599c65446f6","Type":"ContainerStarted","Data":"2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04"}
Nov 04 00:11:53.364815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.364701    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr" event={"ID":"2fbe1cbe-9710-4d1a-9d60-1599c65446f6","Type":"ContainerStarted","Data":"724443d5d218c4336d88fa4ba2e1e8d934827bd528705691f6d38c68f486a074"}
Nov 04 00:11:53.364972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.364952    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:11:53.367143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.367111    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" event={"ID":"324fe1c5-df49-4e36-92d0-f4dccead361a","Type":"ContainerStarted","Data":"5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d"}
Nov 04 00:11:53.413843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.413780    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr" podStartSLOduration=2.413759329 podStartE2EDuration="2.413759329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:53.400930374 +0000 UTC m=+2578.500530518" watchObservedRunningTime="2026-11-04 00:11:53.413759329 +0000 UTC m=+2578.513359473"
Nov 04 00:11:53.462421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.462351    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podStartSLOduration=2.462333553 podStartE2EDuration="2.462333553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:53.452102717 +0000 UTC m=+2578.551702861" watchObservedRunningTime="2026-11-04 00:11:53.462333553 +0000 UTC m=+2578.561933697"
Nov 04 00:11:53.553612 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:53.553503    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9dde98e016dd: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:53.553612 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:53.553530    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:53.553612 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:53.553573    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials podName:2fbe1cbe-9710-4d1a-9d60-1599c65446f6 nodeName:}" failed. No retries permitted until 2026-11-04 00:11:54.053560712 +0000 UTC m=+2579.153160834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials") pod "payload-processing-external-model-78d75b4487-g7gmr" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6") : secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:53.891668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.891578    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:11:53.893404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.893369    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:53.893533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.893427    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:11:53.914990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:53.914950    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"]
Nov 04 00:11:54.046921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.045186    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"]
Nov 04 00:11:54.049956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.049921    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da72753f-vl86x"]
Nov 04 00:11:54.073187 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:54.073157    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9dde98e016dd: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:54.073317 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:54.073194    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:54.073317 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:54.073267    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials podName:2fbe1cbe-9710-4d1a-9d60-1599c65446f6 nodeName:}" failed. No retries permitted until 2026-11-04 00:11:55.07324593 +0000 UTC m=+2580.172846054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials") pod "payload-processing-external-model-78d75b4487-g7gmr" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6") : secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:54.377433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.377399    5074 generic.go:356] "Generic (PLEG): container finished" podID="b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" containerID="bf68460b537ad4bff32cbef2b1a8a04fe205562698a2a8db48a251ed0fd4e1de" exitCode=0
Nov 04 00:11:54.377614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.377496    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs" event={"ID":"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c","Type":"ContainerDied","Data":"bf68460b537ad4bff32cbef2b1a8a04fe205562698a2a8db48a251ed0fd4e1de"}
Nov 04 00:11:54.379181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.379152    5074 generic.go:356] "Generic (PLEG): container finished" podID="83a507c6-2a8a-4e46-b3ec-1dd392193c04" containerID="fe08c857a328e9ab0871c5cb5cc4cdb2f5e08faf6f646c8522223673f2af4c25" exitCode=0
Nov 04 00:11:54.379312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.379233    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb" event={"ID":"83a507c6-2a8a-4e46-b3ec-1dd392193c04","Type":"ContainerDied","Data":"fe08c857a328e9ab0871c5cb5cc4cdb2f5e08faf6f646c8522223673f2af4c25"}
Nov 04 00:11:54.891497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.891460    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:54.891676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:54.891516    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:11:55.041624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.041590    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741f943e-9f7e-4fb1-bbe5-c3acf2e9544e" path="/var/lib/kubelet/pods/741f943e-9f7e-4fb1-bbe5-c3acf2e9544e/volumes"
Nov 04 00:11:55.085478 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:55.085444    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9dde98e016dd: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:55.085478 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:55.085484    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:55.085685 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:55.085553    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials podName:2fbe1cbe-9710-4d1a-9d60-1599c65446f6 nodeName:}" failed. No retries permitted until 2026-11-04 00:11:57.085532643 +0000 UTC m=+2582.185132782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials") pod "payload-processing-external-model-78d75b4487-g7gmr" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6") : secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:55.683055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.683029    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:55.692724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.692692    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z\" (UniqueName: \"kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z\") pod \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") "
Nov 04 00:11:55.692933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.692897    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca\") pod \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") "
Nov 04 00:11:55.693765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.693086    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca\") pod \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\" (UID: \"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c\") "
Nov 04 00:11:55.693765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.693431    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca" pod "b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" (UID: "b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:55.693765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.693705    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:55.695618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.695591    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z" pod "b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" (UID: "b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c"). InnerVolumeSpecName "kube-api-access-brr5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:55.752448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.748846    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca" pod "b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" (UID: "b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:11:55.755502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.755474    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:55.794592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.794555    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc\" (UniqueName: \"kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc\") pod \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") "
Nov 04 00:11:55.794775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.794633    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca\") pod \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") "
Nov 04 00:11:55.794877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.794792    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca\") pod \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\" (UID: \"83a507c6-2a8a-4e46-b3ec-1dd392193c04\") "
Nov 04 00:11:55.795234 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.795213    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brr5z\" (UniqueName: \"kubernetes.io/projected/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-kube-api-access-brr5z\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:55.795234 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.795233    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:55.795849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.795817    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca" pod "83a507c6-2a8a-4e46-b3ec-1dd392193c04" (UID: "83a507c6-2a8a-4e46-b3ec-1dd392193c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:11:55.797321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.797290    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca" pod "83a507c6-2a8a-4e46-b3ec-1dd392193c04" (UID: "83a507c6-2a8a-4e46-b3ec-1dd392193c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:11:55.797517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.797497    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc" pod "83a507c6-2a8a-4e46-b3ec-1dd392193c04" (UID: "83a507c6-2a8a-4e46-b3ec-1dd392193c04"). InnerVolumeSpecName "kube-api-access-2s7jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:11:55.891369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.891336    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:55.891536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.891386    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:11:55.896163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.896138    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s7jc\" (UniqueName: \"kubernetes.io/projected/83a507c6-2a8a-4e46-b3ec-1dd392193c04-kube-api-access-2s7jc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:55.896465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.896166    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:55.896465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:55.896176    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a507c6-2a8a-4e46-b3ec-1dd392193c04-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:11:56.414285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.414245    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs" event={"ID":"b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c","Type":"ContainerDied","Data":"37ba15853158318a0d11d6aefbec0476a32d0a322adfa9936aac076f76122360"}
Nov 04 00:11:56.414610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.414569    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ba15853158318a0d11d6aefbec0476a32d0a322adfa9936aac076f76122360"
Nov 04 00:11:56.414855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.414839    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"
Nov 04 00:11:56.451755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.448153    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb" event={"ID":"83a507c6-2a8a-4e46-b3ec-1dd392193c04","Type":"ContainerDied","Data":"31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59"}
Nov 04 00:11:56.451755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.448192    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"
Nov 04 00:11:56.451755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.448214    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59"
Nov 04 00:11:56.569124 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:56.569085    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a507c6_2a8a_4e46_b3ec_1dd392193c04.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a507c6_2a8a_4e46_b3ec_1dd392193c04.slice/crio-31b1fafd34b6720d62e0034f836e333bd189479ff26e1e8bf9506948b16afa59\": RecentStats: unable to find data in memory cache]"
Nov 04 00:11:56.891354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.891323    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:56.891529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:56.891381    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:11:57.145889 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:57.145425    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9dde98e016dd: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:57.145889 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:57.145468    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr: secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:57.145889 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:57.145527    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials podName:2fbe1cbe-9710-4d1a-9d60-1599c65446f6 nodeName:}" failed. No retries permitted until 2026-11-04 00:12:01.145508231 +0000 UTC m=+2586.245108372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials") pod "payload-processing-external-model-78d75b4487-g7gmr" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6") : secret "agc-cred-9dde98e016dd" not found
Nov 04 00:11:57.891978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:57.891943    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:57.892158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:57.891995    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:11:58.055822 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:11:58.055785    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:11:58.891967 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:58.891925    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:58.892381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:58.891995    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:11:59.291417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:59.291382    5074 scope.go:122] "RemoveContainer" containerID="ae3da5d4c5d0e80f0d4d48c8a320472df0b9c0fffde31c373e356d43e3707291"
Nov 04 00:11:59.305994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:59.305965    5074 scope.go:122] "RemoveContainer" containerID="34d7cf61eb8ec83374a65bf655a4595ff73426288a9e081f580aea6ce7fe0ecb"
Nov 04 00:11:59.323242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:59.323216    5074 scope.go:122] "RemoveContainer" containerID="1b4ff6fb2b2714b5f006450f0597f0873f1f397e9e8feff270d1c4318c27852b"
Nov 04 00:11:59.891716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:59.891677    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Nov 04 00:11:59.891925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:11:59.891731    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Nov 04 00:12:00.388121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.388072    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr" podUID="2fbe1cbe-9710-4d1a-9d60-1599c65446f6" containerName="payload-processing" containerID="cri-o://2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04" gracePeriod=25
Nov 04 00:12:00.697943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.697906    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:12:00.800060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800028    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume\") pod \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") "
Nov 04 00:12:00.800240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800071    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials\") pod \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") "
Nov 04 00:12:00.800240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800088    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume\") pod \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") "
Nov 04 00:12:00.800240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800128    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls\") pod \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\" (UID: \"2fbe1cbe-9710-4d1a-9d60-1599c65446f6\") "
Nov 04 00:12:00.800582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800556    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume" pod "2fbe1cbe-9710-4d1a-9d60-1599c65446f6" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:00.800675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800594    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume" pod "2fbe1cbe-9710-4d1a-9d60-1599c65446f6" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:00.800797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800776    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:00.800873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.800801    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:00.802555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.802526    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls" pod "2fbe1cbe-9710-4d1a-9d60-1599c65446f6" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6"). InnerVolumeSpecName "kube-api-access-j8sls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:00.802668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.802552    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials" pod "2fbe1cbe-9710-4d1a-9d60-1599c65446f6" (UID: "2fbe1cbe-9710-4d1a-9d60-1599c65446f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:00.902121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.902025    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:00.902121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:00.902057    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8sls\" (UniqueName: \"kubernetes.io/projected/2fbe1cbe-9710-4d1a-9d60-1599c65446f6-kube-api-access-j8sls\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:01.520203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.520154    5074 generic.go:356] "Generic (PLEG): container finished" podID="2fbe1cbe-9710-4d1a-9d60-1599c65446f6" containerID="2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04" exitCode=0
Nov 04 00:12:01.521096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.521057    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr" event={"ID":"2fbe1cbe-9710-4d1a-9d60-1599c65446f6","Type":"ContainerDied","Data":"2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04"}
Nov 04 00:12:01.521392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.521358    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr" event={"ID":"2fbe1cbe-9710-4d1a-9d60-1599c65446f6","Type":"ContainerDied","Data":"724443d5d218c4336d88fa4ba2e1e8d934827bd528705691f6d38c68f486a074"}
Nov 04 00:12:01.521757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.521661    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"
Nov 04 00:12:01.527121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.521678    5074 scope.go:122] "RemoveContainer" containerID="2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04"
Nov 04 00:12:01.550996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.550954    5074 scope.go:122] "RemoveContainer" containerID="2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04"
Nov 04 00:12:01.551587 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:12:01.551508    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04\": container with ID starting with 2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04 not found: ID does not exist" containerID="2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04"
Nov 04 00:12:01.551587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.551541    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04"} err="failed to get container status \"2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04\": rpc error: code = NotFound desc = could not find container \"2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04\": container with ID starting with 2125afee5436d30e40a76b1d80d3ea670c9d54dba54dbe403575d37212d08c04 not found: ID does not exist"
Nov 04 00:12:01.574132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.574097    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"]
Nov 04 00:12:01.581534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.581507    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d75b4487-g7gmr"]
Nov 04 00:12:01.684121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.684074    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:12:01.685725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.685698    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:12:01.686849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:01.686824    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:12:03.025394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.025356    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbe1cbe-9710-4d1a-9d60-1599c65446f6" path="/var/lib/kubelet/pods/2fbe1cbe-9710-4d1a-9d60-1599c65446f6/volumes"
Nov 04 00:12:03.777152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.777079    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"]
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.787763    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2fbe1cbe-9710-4d1a-9d60-1599c65446f6" containerName="payload-processing"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.787798    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbe1cbe-9710-4d1a-9d60-1599c65446f6" containerName="payload-processing"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.787825    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" containerName="revoke-keys"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.787834    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" containerName="revoke-keys"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.787884    5074 cpu_manager.go:414] "containerMap: removing container" podUID="83a507c6-2a8a-4e46-b3ec-1dd392193c04" containerName="revoke-keys"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.787893    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a507c6-2a8a-4e46-b3ec-1dd392193c04" containerName="revoke-keys"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.788108    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbe1cbe-9710-4d1a-9d60-1599c65446f6" containerName="payload-processing"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.788123    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" containerName="revoke-keys"
Nov 04 00:12:03.789402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.788137    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a507c6-2a8a-4e46-b3ec-1dd392193c04" containerName="revoke-keys"
Nov 04 00:12:03.910972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.909982    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"]
Nov 04 00:12:03.910972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:03.910113    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.113281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.113176    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.113963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.113933    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.114165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.114144    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2md\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.114397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.114378    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.216257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.216214    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.216590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.216566    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2md\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.216835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.216807    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.217221 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.217196    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.224780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.224676    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.224780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.224697    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.253355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.253322    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.501772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.478933    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2md\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:04.597759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:04.597707    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:05.264930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:05.264894    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"]
Nov 04 00:12:05.266396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:05.266364    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"]
Nov 04 00:12:05.736493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:05.721809    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:05.885533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:05.885479    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"]
Nov 04 00:12:05.976108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:05.976067    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm" event={"ID":"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415","Type":"ContainerStarted","Data":"21bffb9ceb9829113a36fea8d2108c45bafafecc8db8bb4022eee0ab5e1c9b63"}
Nov 04 00:12:06.052542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.052499    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"]
Nov 04 00:12:06.056881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.056839    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7a465d-877b8"]
Nov 04 00:12:06.437651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.437621    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"]
Nov 04 00:12:06.442628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.442608    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.475610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.475104    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"]
Nov 04 00:12:06.600768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.585444    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md685\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.600768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.585584    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.600768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.585667    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.600768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.585758    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.692315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.692223    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.692617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.692589    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.692918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.692901    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md685\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.698924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.698893    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.702942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.700061    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.707470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.707439    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.713457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.713427    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.771898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.764705    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md685\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685\") pod \"payload-processing-external-model-6bd64bd569-vw4cq\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") " pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:06.771898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.766817    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"]
Nov 04 00:12:06.776454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.776426    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.778617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.778586    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"]
Nov 04 00:12:06.905794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.905735    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.905845    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.905932    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.906033    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.906065    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxz6n\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.906135    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.906195    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.906307    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.906438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.906342    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:06.986247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.986158    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm" event={"ID":"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415","Type":"ContainerStarted","Data":"ea6069e7faa24998b33fffa8c3ff7d5e87603acaf5f76a1de4ae94e21281893e"}
Nov 04 00:12:06.986489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:06.986468    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:07.007931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.007899    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.007989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008020    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008125    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008242    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008276    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008299    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008367    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008422    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxz6n\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008474    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008504    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008570    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.008857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.008830    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.010860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.010838    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.010860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.010844    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.020355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.020328    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606" path="/var/lib/kubelet/pods/ff9a2a3b-077b-4b5a-9e4f-1b6cd1409606/volumes"
Nov 04 00:12:07.058092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.058050    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:07.234417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.234361    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm" podStartSLOduration=4.234347689 podStartE2EDuration="4.234347689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:07.224573466 +0000 UTC m=+2592.324173611" watchObservedRunningTime="2026-11-04 00:12:07.234347689 +0000 UTC m=+2592.333947833"
Nov 04 00:12:07.238265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.238200    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.238470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.238446    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxz6n\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8s5gs\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.358449 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:07.358411    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e485bf1_7f02_4ed0_b5f0_f12fdd46be26.slice/crio-4fb9c8120f2283d0951ee11df4f461676928a7d1e3f5954eb4519b0680cc8950 WatchSource:0}: Error finding container 4fb9c8120f2283d0951ee11df4f461676928a7d1e3f5954eb4519b0680cc8950: Status 404 returned error can't find the container with id 4fb9c8120f2283d0951ee11df4f461676928a7d1e3f5954eb4519b0680cc8950
Nov 04 00:12:07.367099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.367065    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"]
Nov 04 00:12:07.406365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.406337    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:07.719057 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:07.719023    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16665b8_d193_41b7_a0b4_32108cbf6a6f.slice/crio-4ca6af0a79a511e394da5c4780ab3e04ca0d2908d218f79cef392db3d0a0d70c WatchSource:0}: Error finding container 4ca6af0a79a511e394da5c4780ab3e04ca0d2908d218f79cef392db3d0a0d70c: Status 404 returned error can't find the container with id 4ca6af0a79a511e394da5c4780ab3e04ca0d2908d218f79cef392db3d0a0d70c
Nov 04 00:12:07.723008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.722866    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"]
Nov 04 00:12:07.725941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.725802    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:12:07.725941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.725873    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:12:07.725941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.725903    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:12:07.974906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.974871    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"]
Nov 04 00:12:07.980891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:07.980152    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.023575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.023080    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"agc-cred-66735f7932d4\""
Nov 04 00:12:08.026766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.026005    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbbm\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.026766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.026139    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.026766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.026187    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.026766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.026217    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.041419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.041392    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-71b2f3\""
Nov 04 00:12:08.065874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.052004    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"routing-overlay\""
Nov 04 00:12:08.065874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.052105    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"kube-root-ca.crt\""
Nov 04 00:12:08.065874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.052168    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"openshift-service-ca.crt\""
Nov 04 00:12:08.065874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.052220    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"payload-processing-external-model-e2e--5c4b218e-dockercfg-b5mgl\""
Nov 04 00:12:08.075763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.074729    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq" event={"ID":"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26","Type":"ContainerStarted","Data":"724888e1b1b22ecd61f700be128b812986476096883a7ffbc504f140f740d12b"}
Nov 04 00:12:08.075763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.074816    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq" event={"ID":"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26","Type":"ContainerStarted","Data":"4fb9c8120f2283d0951ee11df4f461676928a7d1e3f5954eb4519b0680cc8950"}
Nov 04 00:12:08.076204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.076182    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:08.077356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.077334    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"]
Nov 04 00:12:08.079690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.079666    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" event={"ID":"d16665b8-d193-41b7-a0b4-32108cbf6a6f","Type":"ContainerStarted","Data":"6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57"}
Nov 04 00:12:08.079892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.079873    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" event={"ID":"d16665b8-d193-41b7-a0b4-32108cbf6a6f","Type":"ContainerStarted","Data":"4ca6af0a79a511e394da5c4780ab3e04ca0d2908d218f79cef392db3d0a0d70c"}
Nov 04 00:12:08.144763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.142108    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.144763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.142191    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.144763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.142216    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.144763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.142524    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbbm\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.156606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.156564    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.157342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.157315    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.166124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.166089    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbbm\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.167098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.167073    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.174949 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.174883    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq" podStartSLOduration=2.1748622810000002 podStartE2EDuration="2.174862281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:08.139353883 +0000 UTC m=+2593.238954031" watchObservedRunningTime="2026-11-04 00:12:08.174862281 +0000 UTC m=+2593.274462425"
Nov 04 00:12:08.260594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.260535    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podStartSLOduration=2.260519273 podStartE2EDuration="2.260519273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:08.233266436 +0000 UTC m=+2593.332866579" watchObservedRunningTime="2026-11-04 00:12:08.260519273 +0000 UTC m=+2593.360119417"
Nov 04 00:12:08.326527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.326442    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:08.407236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.406940    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:08.413401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.410336    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:08.413401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.410387    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:08.637786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:08.637617    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"]
Nov 04 00:12:08.639997 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:08.639879    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4fe9b8_66aa_4ff8_91b7_908b164c79bf.slice/crio-e44bb52db6f83cb2982ca136ea7af5addca6eee6b319f280a1f85b63e4a74250 WatchSource:0}: Error finding container e44bb52db6f83cb2982ca136ea7af5addca6eee6b319f280a1f85b63e4a74250: Status 404 returned error can't find the container with id e44bb52db6f83cb2982ca136ea7af5addca6eee6b319f280a1f85b63e4a74250
Nov 04 00:12:09.090090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:09.090039    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h" event={"ID":"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf","Type":"ContainerStarted","Data":"fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288"}
Nov 04 00:12:09.090550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:09.090108    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h" event={"ID":"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf","Type":"ContainerStarted","Data":"e44bb52db6f83cb2982ca136ea7af5addca6eee6b319f280a1f85b63e4a74250"}
Nov 04 00:12:09.090550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:09.090146    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:09.153164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:09.153110    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h" podStartSLOduration=2.153094474 podStartE2EDuration="2.153094474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:09.147040267 +0000 UTC m=+2594.246640410" watchObservedRunningTime="2026-11-04 00:12:09.153094474 +0000 UTC m=+2594.252694618"
Nov 04 00:12:09.410817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:09.410160    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:09.410817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:09.410228    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:10.145306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.145266    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"]
Nov 04 00:12:10.150416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.150391    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.164887 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.164861    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"agc-cred-c54e0cfe3210\""
Nov 04 00:12:10.171483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.171413    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"]
Nov 04 00:12:10.173334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.173255    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-06ade4\""
Nov 04 00:12:10.173334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.173259    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"routing-overlay\""
Nov 04 00:12:10.173334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.173306    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"openshift-service-ca.crt\""
Nov 04 00:12:10.173334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.173314    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"payload-processing-external-model-e2e--f0c21228-dockercfg-c5cmt\""
Nov 04 00:12:10.175399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.175364    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"kube-root-ca.crt\""
Nov 04 00:12:10.182099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.182069    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.183632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.183611    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.184752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.184631    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.184752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.184706    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.267106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.267049    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk" podUID="33ee3776-17e7-4bbd-9c4d-96c7d633bceb" containerName="payload-processing" containerID="cri-o://c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d" gracePeriod=25
Nov 04 00:12:10.286761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.286088    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.286761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.286682    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.286761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.286758    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.287103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.286838    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.287976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.287596    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.289272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.288374    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.294358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.294335    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.381373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.381337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.411751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.411654    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:10.412005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.411974    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:10.483200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.481844    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:10.572121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.525687    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"]
Nov 04 00:12:10.584525 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.584496    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"]
Nov 04 00:12:10.584903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.584798    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.634434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.634396    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8895cb\""
Nov 04 00:12:10.635446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.635406    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"routing-overlay\""
Nov 04 00:12:10.637693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.637658    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"kube-root-ca.crt\""
Nov 04 00:12:10.680136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.637937    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"openshift-service-ca.crt\""
Nov 04 00:12:10.680289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.640112    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.680467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.680442    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2xs\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.680589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.680576    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.680715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.680706    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.681789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.640253    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-86bb370d5296\""
Nov 04 00:12:10.681789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.640348    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-e2e--9366bb2e-dockercfg-z9lrg\""
Nov 04 00:12:10.788126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.783185    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.788126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.787460    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2xs\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.788126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.787549    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.788126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.787653    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.794277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.791771    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.794883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.794693    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.888075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.862846    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2xs\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.920787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.914416    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:10.989040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:10.987935    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:11.043354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.042940    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:12:11.143863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.143826    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials\") pod \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") "
Nov 04 00:12:11.144036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.143932    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh\") pod \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") "
Nov 04 00:12:11.144161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.144144    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume\") pod \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") "
Nov 04 00:12:11.144302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.144285    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume\") pod \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\" (UID: \"33ee3776-17e7-4bbd-9c4d-96c7d633bceb\") "
Nov 04 00:12:11.152915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.152607    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume" pod "33ee3776-17e7-4bbd-9c4d-96c7d633bceb" (UID: "33ee3776-17e7-4bbd-9c4d-96c7d633bceb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:11.153549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.153511    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume" pod "33ee3776-17e7-4bbd-9c4d-96c7d633bceb" (UID: "33ee3776-17e7-4bbd-9c4d-96c7d633bceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:11.154868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.154679    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh" pod "33ee3776-17e7-4bbd-9c4d-96c7d633bceb" (UID: "33ee3776-17e7-4bbd-9c4d-96c7d633bceb"). InnerVolumeSpecName "kube-api-access-zrnfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:11.158869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.158823    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials" pod "33ee3776-17e7-4bbd-9c4d-96c7d633bceb" (UID: "33ee3776-17e7-4bbd-9c4d-96c7d633bceb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:11.165594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.165559    5074 generic.go:356] "Generic (PLEG): container finished" podID="33ee3776-17e7-4bbd-9c4d-96c7d633bceb" containerID="c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d" exitCode=0
Nov 04 00:12:11.165832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.165809    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk" event={"ID":"33ee3776-17e7-4bbd-9c4d-96c7d633bceb","Type":"ContainerDied","Data":"c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d"}
Nov 04 00:12:11.165919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.165868    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk" event={"ID":"33ee3776-17e7-4bbd-9c4d-96c7d633bceb","Type":"ContainerDied","Data":"6f0d8379bc780176058b7370ea337eaa16455bd3a9b84f52a269941cce3fde80"}
Nov 04 00:12:11.165919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.165899    5074 scope.go:122] "RemoveContainer" containerID="c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d"
Nov 04 00:12:11.166097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.166080    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"
Nov 04 00:12:11.186972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.186951    5074 scope.go:122] "RemoveContainer" containerID="c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d"
Nov 04 00:12:11.187271 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:12:11.187249    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d\": container with ID starting with c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d not found: ID does not exist" containerID="c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d"
Nov 04 00:12:11.187354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.187279    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d"} err="failed to get container status \"c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d\": rpc error: code = NotFound desc = could not find container \"c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d\": container with ID starting with c5194b4ec8f7ec3a669c9f59252b763c352c469c8805289d506d281858efdd4d not found: ID does not exist"
Nov 04 00:12:11.227888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.227855    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"]
Nov 04 00:12:11.248506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.248479    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"]
Nov 04 00:12:11.252155 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:11.249967    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332c0e88_3f29_4d3a_a2a0_e8abbd478a38.slice/crio-4f9c41379e3431721926f8956a45f44feff8ab4409f92599e8b62c2cd2a740dc WatchSource:0}: Error finding container 4f9c41379e3431721926f8956a45f44feff8ab4409f92599e8b62c2cd2a740dc: Status 404 returned error can't find the container with id 4f9c41379e3431721926f8956a45f44feff8ab4409f92599e8b62c2cd2a740dc
Nov 04 00:12:11.253711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.253651    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:11.253711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.253692    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrnfh\" (UniqueName: \"kubernetes.io/projected/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-kube-api-access-zrnfh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:11.253833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.253714    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:11.253833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.253731    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ee3776-17e7-4bbd-9c4d-96c7d633bceb-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:11.258788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.256977    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-7cc8pb9mk"]
Nov 04 00:12:11.368667 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:11.368626    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b9fe83_fb50_412d_8c8c_d525c73c2be3.slice/crio-351248bd8df8f5672742d012769aef9c4b52bcef1f68c24b45a197b24f2c387f WatchSource:0}: Error finding container 351248bd8df8f5672742d012769aef9c4b52bcef1f68c24b45a197b24f2c387f: Status 404 returned error can't find the container with id 351248bd8df8f5672742d012769aef9c4b52bcef1f68c24b45a197b24f2c387f
Nov 04 00:12:11.380541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.380504    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"]
Nov 04 00:12:11.425763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.410081    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:11.425763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.410138    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:11.987384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:11.987319    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm" podUID="1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" containerName="payload-processing" containerID="cri-o://ea6069e7faa24998b33fffa8c3ff7d5e87603acaf5f76a1de4ae94e21281893e" gracePeriod=25
Nov 04 00:12:12.182660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.182608    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9" event={"ID":"332c0e88-3f29-4d3a-a2a0-e8abbd478a38","Type":"ContainerStarted","Data":"e8bb90c3ce10807bcf8bae947f980ae44ca20d8625762f37ba8536277c42aad8"}
Nov 04 00:12:12.183481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.183455    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9" event={"ID":"332c0e88-3f29-4d3a-a2a0-e8abbd478a38","Type":"ContainerStarted","Data":"4f9c41379e3431721926f8956a45f44feff8ab4409f92599e8b62c2cd2a740dc"}
Nov 04 00:12:12.183665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.183640    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:12.184848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.184822    5074 generic.go:356] "Generic (PLEG): container finished" podID="1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" containerID="ea6069e7faa24998b33fffa8c3ff7d5e87603acaf5f76a1de4ae94e21281893e" exitCode=0
Nov 04 00:12:12.185094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.184934    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm" event={"ID":"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415","Type":"ContainerDied","Data":"ea6069e7faa24998b33fffa8c3ff7d5e87603acaf5f76a1de4ae94e21281893e"}
Nov 04 00:12:12.187784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.187764    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x" event={"ID":"97b9fe83-fb50-412d-8c8c-d525c73c2be3","Type":"ContainerStarted","Data":"f0b0d04e7dfedd3e8021b83ec0b4976924ef58181f5fd6b788f04caf23608b45"}
Nov 04 00:12:12.187857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.187801    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x" event={"ID":"97b9fe83-fb50-412d-8c8c-d525c73c2be3","Type":"ContainerStarted","Data":"351248bd8df8f5672742d012769aef9c4b52bcef1f68c24b45a197b24f2c387f"}
Nov 04 00:12:12.249827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.249682    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9" podStartSLOduration=2.249663043 podStartE2EDuration="2.249663043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:12.246898547 +0000 UTC m=+2597.346498691" watchObservedRunningTime="2026-11-04 00:12:12.249663043 +0000 UTC m=+2597.349263190"
Nov 04 00:12:12.282773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.282496    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x" podStartSLOduration=2.282481678 podStartE2EDuration="2.282481678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:12.278423757 +0000 UTC m=+2597.378023901" watchObservedRunningTime="2026-11-04 00:12:12.282481678 +0000 UTC m=+2597.382081821"
Nov 04 00:12:12.379026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.378999    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:12.435762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.429140    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:12.435762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.429205    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:12.471524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.471486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md\") pod \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") "
Nov 04 00:12:12.471699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.471642    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume\") pod \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") "
Nov 04 00:12:12.471805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.471788    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume\") pod \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") "
Nov 04 00:12:12.472021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.471994    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume" pod "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" (UID: "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:12.472111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.472087    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials\") pod \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\" (UID: \"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415\") "
Nov 04 00:12:12.472183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.472168    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume" pod "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" (UID: "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:12.472554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.472536    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:12.472554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.472554    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:12.474099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.474075    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md" pod "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" (UID: "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415"). InnerVolumeSpecName "kube-api-access-bl2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:12.476899 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.476873    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials" pod "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" (UID: "1c54b4e6-81f9-42b1-8f15-b4cb30a5b415"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:12.574316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.574234    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:12.574316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:12.574263    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2md\" (UniqueName: \"kubernetes.io/projected/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415-kube-api-access-bl2md\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:13.040610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.040571    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ee3776-17e7-4bbd-9c4d-96c7d633bceb" path="/var/lib/kubelet/pods/33ee3776-17e7-4bbd-9c4d-96c7d633bceb/volumes"
Nov 04 00:12:13.197654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.197624    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"
Nov 04 00:12:13.198181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.197648    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm" event={"ID":"1c54b4e6-81f9-42b1-8f15-b4cb30a5b415","Type":"ContainerDied","Data":"21bffb9ceb9829113a36fea8d2108c45bafafecc8db8bb4022eee0ab5e1c9b63"}
Nov 04 00:12:13.198181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.197717    5074 scope.go:122] "RemoveContainer" containerID="ea6069e7faa24998b33fffa8c3ff7d5e87603acaf5f76a1de4ae94e21281893e"
Nov 04 00:12:13.214762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.198774    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:13.250900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.250870    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"]
Nov 04 00:12:13.335294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.335199    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-859cl82gm"]
Nov 04 00:12:13.409502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.409069    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:13.409502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:13.409131    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:14.407871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:14.407826    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:14.408402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:14.407894    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:15.023647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:15.023606    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" path="/var/lib/kubelet/pods/1c54b4e6-81f9-42b1-8f15-b4cb30a5b415/volumes"
Nov 04 00:12:15.408144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:15.407432    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Nov 04 00:12:15.408144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:15.407507    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Nov 04 00:12:16.924413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:16.924365    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:16.925546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:16.925510    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:16.926483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:16.926456    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:12:18.041155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:18.041111    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"]
Nov 04 00:12:18.047989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:18.047952    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c34bc3e-tqkgh"]
Nov 04 00:12:19.022577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:19.022540    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5f4282-4d1a-467e-b3c5-e0c361968222" path="/var/lib/kubelet/pods/fa5f4282-4d1a-467e-b3c5-e0c361968222/volumes"
Nov 04 00:12:20.106441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:20.106408    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:12:20.106932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:20.106495    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:12:20.189152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:20.189120    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"]
Nov 04 00:12:22.987293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.987258    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"]
Nov 04 00:12:22.988128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.988109    5074 cpu_manager.go:414] "containerMap: removing container" podUID="33ee3776-17e7-4bbd-9c4d-96c7d633bceb" containerName="payload-processing"
Nov 04 00:12:22.988192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.988131    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ee3776-17e7-4bbd-9c4d-96c7d633bceb" containerName="payload-processing"
Nov 04 00:12:22.988192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.988160    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" containerName="payload-processing"
Nov 04 00:12:22.988192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.988168    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" containerName="payload-processing"
Nov 04 00:12:22.988369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.988355    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ee3776-17e7-4bbd-9c4d-96c7d633bceb" containerName="payload-processing"
Nov 04 00:12:22.988408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.988379    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c54b4e6-81f9-42b1-8f15-b4cb30a5b415" containerName="payload-processing"
Nov 04 00:12:22.992665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:22.992642    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.004997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.004972    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"kube-root-ca.crt\""
Nov 04 00:12:23.005325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.005303    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-fa0d0a\""
Nov 04 00:12:23.007223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.007201    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"routing-overlay\""
Nov 04 00:12:23.007710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.007689    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"openshift-service-ca.crt\""
Nov 04 00:12:23.009662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.009640    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"]
Nov 04 00:12:23.029997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.029968    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"agc-cred-b5e0426aec0a\""
Nov 04 00:12:23.149227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.149194    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thf6z\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.149463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.149233    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.149463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.149419    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.149637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.149479    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.251176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.250725    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thf6z\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.251492 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.251446    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.251804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.251782    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.252007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.251989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.253325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.253302    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.255909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.255884    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.259965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.259921    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.537851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.533063    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thf6z\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.538584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.538554    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"]
Nov 04 00:12:23.551215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.551074    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.555370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.555335    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"]
Nov 04 00:12:23.571063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.571035    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"routing-overlay\""
Nov 04 00:12:23.571202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.571039    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"kube-root-ca.crt\""
Nov 04 00:12:23.571202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.571138    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-e2e--5d63ff17-dockercfg-q6tbd\""
Nov 04 00:12:23.571598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.571577    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2697a0\""
Nov 04 00:12:23.573532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.573507    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-c52a4485c26d\""
Nov 04 00:12:23.573637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.573542    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"openshift-service-ca.crt\""
Nov 04 00:12:23.610787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.610730    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:23.658287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.658243    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.658459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.658387    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclbp\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.658565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.658543    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.658714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.658697    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.761065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.760456    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.761065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.760538    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclbp\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.761065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.760626    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.761065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.760709    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.761501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.761485    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.762158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.762093    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.773484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.770242    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.834480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.834400    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclbp\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.876601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.876568    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:23.904073 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:23.904047    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5be7f02_9455_4446_b8b7_1f82ca97249f.slice/crio-448bb0ce53649fdaad9bec58c0aefe704ffa37f2fc0c526395b046b43cfdb5fd WatchSource:0}: Error finding container 448bb0ce53649fdaad9bec58c0aefe704ffa37f2fc0c526395b046b43cfdb5fd: Status 404 returned error can't find the container with id 448bb0ce53649fdaad9bec58c0aefe704ffa37f2fc0c526395b046b43cfdb5fd
Nov 04 00:12:23.906705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:23.906673    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"]
Nov 04 00:12:24.170342 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:12:24.170291    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81887362_c237_476a_a338_8c325256dc90.slice/crio-702b80097dd474ad89236394730863400305760359f4b4d19901aedacbb0119b WatchSource:0}: Error finding container 702b80097dd474ad89236394730863400305760359f4b4d19901aedacbb0119b: Status 404 returned error can't find the container with id 702b80097dd474ad89236394730863400305760359f4b4d19901aedacbb0119b
Nov 04 00:12:24.175452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.175372    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"]
Nov 04 00:12:24.234170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.233260    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:12:24.240076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.239147    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:12:24.407026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.406980    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv" event={"ID":"81887362-c237-476a-a338-8c325256dc90","Type":"ContainerStarted","Data":"5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13"}
Nov 04 00:12:24.407228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.407057    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv" event={"ID":"81887362-c237-476a-a338-8c325256dc90","Type":"ContainerStarted","Data":"702b80097dd474ad89236394730863400305760359f4b4d19901aedacbb0119b"}
Nov 04 00:12:24.408236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.408203    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:24.412012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.411986    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd" event={"ID":"f5be7f02-9455-4446-b8b7-1f82ca97249f","Type":"ContainerStarted","Data":"58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166"}
Nov 04 00:12:24.412167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.412152    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd" event={"ID":"f5be7f02-9455-4446-b8b7-1f82ca97249f","Type":"ContainerStarted","Data":"448bb0ce53649fdaad9bec58c0aefe704ffa37f2fc0c526395b046b43cfdb5fd"}
Nov 04 00:12:24.412853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.412833    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:24.475879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.475818    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv" podStartSLOduration=1.475804441 podStartE2EDuration="1.475804441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:24.472199432 +0000 UTC m=+2609.571799578" watchObservedRunningTime="2026-11-04 00:12:24.475804441 +0000 UTC m=+2609.575404584"
Nov 04 00:12:24.514977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:24.514912    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd" podStartSLOduration=2.514892702 podStartE2EDuration="2.514892702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:24.502661831 +0000 UTC m=+2609.602261974" watchObservedRunningTime="2026-11-04 00:12:24.514892702 +0000 UTC m=+2609.614492847"
Nov 04 00:12:25.190099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:25.190046    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g" podUID="846b7c16-9dcc-493e-ba2a-0ee844093a4f" containerName="payload-processing" containerID="cri-o://4572a3e8c1b1d50f37334ad8e7301c00214c1afb434af196d8ed70f7514926bc" gracePeriod=25
Nov 04 00:12:25.476291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:25.474304    5074 generic.go:356] "Generic (PLEG): container finished" podID="846b7c16-9dcc-493e-ba2a-0ee844093a4f" containerID="4572a3e8c1b1d50f37334ad8e7301c00214c1afb434af196d8ed70f7514926bc" exitCode=0
Nov 04 00:12:25.476291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:25.475463    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g" event={"ID":"846b7c16-9dcc-493e-ba2a-0ee844093a4f","Type":"ContainerDied","Data":"4572a3e8c1b1d50f37334ad8e7301c00214c1afb434af196d8ed70f7514926bc"}
Nov 04 00:12:26.192774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.187121    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:12:26.209595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.209560    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume\") pod \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") "
Nov 04 00:12:26.211425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.211399    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk\") pod \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") "
Nov 04 00:12:26.211698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.211678    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials\") pod \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") "
Nov 04 00:12:26.215906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.215882    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume\") pod \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\" (UID: \"846b7c16-9dcc-493e-ba2a-0ee844093a4f\") "
Nov 04 00:12:26.249813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.210202    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume" pod "846b7c16-9dcc-493e-ba2a-0ee844093a4f" (UID: "846b7c16-9dcc-493e-ba2a-0ee844093a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:26.274768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.273630    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk" pod "846b7c16-9dcc-493e-ba2a-0ee844093a4f" (UID: "846b7c16-9dcc-493e-ba2a-0ee844093a4f"). InnerVolumeSpecName "kube-api-access-sqfpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:26.275167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.275134    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume" pod "846b7c16-9dcc-493e-ba2a-0ee844093a4f" (UID: "846b7c16-9dcc-493e-ba2a-0ee844093a4f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:12:26.355769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.329525    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials" pod "846b7c16-9dcc-493e-ba2a-0ee844093a4f" (UID: "846b7c16-9dcc-493e-ba2a-0ee844093a4f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:12:26.398793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.356104    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:26.398793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.356149    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfpk\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-kube-api-access-sqfpk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:26.398793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.356166    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/846b7c16-9dcc-493e-ba2a-0ee844093a4f-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:26.398793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.356181    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/846b7c16-9dcc-493e-ba2a-0ee844093a4f-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:12:26.532060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.531809    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"
Nov 04 00:12:26.544780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.541805    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c4895777b-jwv7g" event={"ID":"846b7c16-9dcc-493e-ba2a-0ee844093a4f","Type":"ContainerDied","Data":"41d036137aa4724a24a558afd9ecb5540beedbc3c23829f3ee06e7196a2c1554"}
Nov 04 00:12:26.544780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.541905    5074 scope.go:122] "RemoveContainer" containerID="4572a3e8c1b1d50f37334ad8e7301c00214c1afb434af196d8ed70f7514926bc"
Nov 04 00:12:27.008672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:26.998820    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:27.046937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.046883    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 04 00:12:27.149866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.139015    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"]
Nov 04 00:12:27.155141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.155104    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c4895777b-jwv7g"]
Nov 04 00:12:27.341723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.339990    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 04 00:12:27.377274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.376820    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:27.377274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.376899    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:27.421529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:27.421487    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:28.026196 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:12:28.023448    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:12:29.026877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:29.026840    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846b7c16-9dcc-493e-ba2a-0ee844093a4f" path="/var/lib/kubelet/pods/846b7c16-9dcc-493e-ba2a-0ee844093a4f/volumes"
Nov 04 00:12:36.535263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:36.535231    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:12:36.546326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:36.546301    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:12:57.036280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:57.036247    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"]
Nov 04 00:12:57.041260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:57.041231    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5020b25a-2kskm"]
Nov 04 00:12:58.024011 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:12:58.023970    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:12:59.021468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:59.021428    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb" path="/var/lib/kubelet/pods/b410d1fd-7f3a-4ff9-a6aa-5942e6ac2fbb/volumes"
Nov 04 00:12:59.412952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:59.412860    5074 scope.go:122] "RemoveContainer" containerID="ed35d696b73092788506dfeeedc1347d1e251e3efa67ed4428ce7a6e29920e08"
Nov 04 00:12:59.424300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:12:59.424276    5074 scope.go:122] "RemoveContainer" containerID="d264372e55a9630f125cff3c6844a56f8aa263391e187de96d5d4a8c2b104fac"
Nov 04 00:13:18.233657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.233612    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"]
Nov 04 00:13:18.234502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.234482    5074 cpu_manager.go:414] "containerMap: removing container" podUID="846b7c16-9dcc-493e-ba2a-0ee844093a4f" containerName="payload-processing"
Nov 04 00:13:18.234579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.234502    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846b7c16-9dcc-493e-ba2a-0ee844093a4f" containerName="payload-processing"
Nov 04 00:13:18.234673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.234660    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="846b7c16-9dcc-493e-ba2a-0ee844093a4f" containerName="payload-processing"
Nov 04 00:13:18.240858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.240816    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.248094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.248067    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"]
Nov 04 00:13:18.249302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.248954    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:13:18.249302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.249102    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:13:18.298079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.298036    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.298258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.298181    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwk9c\" (UniqueName: \"kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.298462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.298427    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.399757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.399703    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwk9c\" (UniqueName: \"kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.400019 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.399999    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.400120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.400105    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.400875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.400847    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.402960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.402936    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.456060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.456006    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwk9c\" (UniqueName: \"kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c\") pod \"maas-api-revoke-sub-3e135e17-c998q\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.566802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.566690    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:18.855656 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:18.855613    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23d47eb_235c_4547_b6ba_fc5b6059094a.slice/crio-6c3ea7feb05ef900031d6b08acc1746cec0d7cf32361eb0818cc43b125f5ec3a WatchSource:0}: Error finding container 6c3ea7feb05ef900031d6b08acc1746cec0d7cf32361eb0818cc43b125f5ec3a: Status 404 returned error can't find the container with id 6c3ea7feb05ef900031d6b08acc1746cec0d7cf32361eb0818cc43b125f5ec3a
Nov 04 00:13:18.858803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:18.858772    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"]
Nov 04 00:13:19.271613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:19.271578    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q" event={"ID":"f23d47eb-235c-4547-b6ba-fc5b6059094a","Type":"ContainerStarted","Data":"6c3ea7feb05ef900031d6b08acc1746cec0d7cf32361eb0818cc43b125f5ec3a"}
Nov 04 00:13:20.284663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:20.284629    5074 generic.go:356] "Generic (PLEG): container finished" podID="f23d47eb-235c-4547-b6ba-fc5b6059094a" containerID="609e0d2816c98d761a68013dc6a6b6293ac5dec2f8abc1a256ede6e017ffb779" exitCode=0
Nov 04 00:13:20.285088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:20.284748    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q" event={"ID":"f23d47eb-235c-4547-b6ba-fc5b6059094a","Type":"ContainerDied","Data":"609e0d2816c98d761a68013dc6a6b6293ac5dec2f8abc1a256ede6e017ffb779"}
Nov 04 00:13:21.462359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.462333    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:21.560615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.560527    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c\" (UniqueName: \"kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c\") pod \"f23d47eb-235c-4547-b6ba-fc5b6059094a\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") "
Nov 04 00:13:21.560821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.560630    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca\") pod \"f23d47eb-235c-4547-b6ba-fc5b6059094a\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") "
Nov 04 00:13:21.560821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.560731    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca\") pod \"f23d47eb-235c-4547-b6ba-fc5b6059094a\" (UID: \"f23d47eb-235c-4547-b6ba-fc5b6059094a\") "
Nov 04 00:13:21.561094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.561071    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca" pod "f23d47eb-235c-4547-b6ba-fc5b6059094a" (UID: "f23d47eb-235c-4547-b6ba-fc5b6059094a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:21.561363 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.561341    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:21.562706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.562679    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c" pod "f23d47eb-235c-4547-b6ba-fc5b6059094a" (UID: "f23d47eb-235c-4547-b6ba-fc5b6059094a"). InnerVolumeSpecName "kube-api-access-rwk9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:21.562837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.562727    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca" pod "f23d47eb-235c-4547-b6ba-fc5b6059094a" (UID: "f23d47eb-235c-4547-b6ba-fc5b6059094a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:13:21.662965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.662932    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwk9c\" (UniqueName: \"kubernetes.io/projected/f23d47eb-235c-4547-b6ba-fc5b6059094a-kube-api-access-rwk9c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:21.662965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:21.662962    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d47eb-235c-4547-b6ba-fc5b6059094a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:22.302556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:22.302524    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"
Nov 04 00:13:22.302756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:22.302566    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q" event={"ID":"f23d47eb-235c-4547-b6ba-fc5b6059094a","Type":"ContainerDied","Data":"6c3ea7feb05ef900031d6b08acc1746cec0d7cf32361eb0818cc43b125f5ec3a"}
Nov 04 00:13:22.302756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:22.302616    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c3ea7feb05ef900031d6b08acc1746cec0d7cf32361eb0818cc43b125f5ec3a"
Nov 04 00:13:28.023514 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:28.023465    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:13:29.460452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.460414    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"]
Nov 04 00:13:29.461457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.461433    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f23d47eb-235c-4547-b6ba-fc5b6059094a" containerName="revoke-keys"
Nov 04 00:13:29.461457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.461458    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23d47eb-235c-4547-b6ba-fc5b6059094a" containerName="revoke-keys"
Nov 04 00:13:29.461664 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.461649    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23d47eb-235c-4547-b6ba-fc5b6059094a" containerName="revoke-keys"
Nov 04 00:13:29.465704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.465683    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.474046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.474018    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"]
Nov 04 00:13:29.484563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.484536    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.484694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.484666    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.484890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.484843    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.485159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.485131    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffbd\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.586278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.586241    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffbd\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.586470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.586293    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.586470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.586354    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.586470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.586430    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.587037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.587010    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.587161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.587128    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.588962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.588934    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.599447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.599428    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffbd\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd\") pod \"payload-processing-external-model-56f897f846-rzrdx\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:29.799140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:29.798830    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:30.134522 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:30.134490    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c0f7db_e15b_4cfc_894d_832cfdd87b61.slice/crio-ffd57e699792b51b7f748a9199b0b5ee0c1d57fb0e755ef9bf7c073c60b17ec3 WatchSource:0}: Error finding container ffd57e699792b51b7f748a9199b0b5ee0c1d57fb0e755ef9bf7c073c60b17ec3: Status 404 returned error can't find the container with id ffd57e699792b51b7f748a9199b0b5ee0c1d57fb0e755ef9bf7c073c60b17ec3
Nov 04 00:13:30.136762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:30.136720    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"]
Nov 04 00:13:30.387995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:30.387884    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx" event={"ID":"e0c0f7db-e15b-4cfc-894d-832cfdd87b61","Type":"ContainerStarted","Data":"8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1"}
Nov 04 00:13:30.387995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:30.387960    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx" event={"ID":"e0c0f7db-e15b-4cfc-894d-832cfdd87b61","Type":"ContainerStarted","Data":"ffd57e699792b51b7f748a9199b0b5ee0c1d57fb0e755ef9bf7c073c60b17ec3"}
Nov 04 00:13:30.387995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:30.387993    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:31.972006 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:31.971935    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx" podStartSLOduration=2.971917105 podStartE2EDuration="2.971917105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:30.41364308 +0000 UTC m=+2675.513243225" watchObservedRunningTime="2026-11-04 00:13:31.971917105 +0000 UTC m=+2677.071517249"
Nov 04 00:13:31.975808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:31.975770    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"]
Nov 04 00:13:32.012552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.012519    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"]
Nov 04 00:13:32.018951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.018923    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.024562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.024530    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"]
Nov 04 00:13:32.027028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.026132    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f0ff239bd626\""
Nov 04 00:13:32.158252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.158207    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.158613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.158590    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.158889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.158871    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxnm2\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.159239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.159222    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.260868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.260778    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.261029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.260911    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.261029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.260989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.261119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.261062    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnm2\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.261835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.261804    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.261971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.261807    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.263681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.263653    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.274320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.274290    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnm2\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2\") pod \"payload-processing-external-model-9d7d778bd-6qrfb\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.369925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.369888    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:32.538847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.536153    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"]
Nov 04 00:13:32.579768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.571788    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"]
Nov 04 00:13:32.760128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.758346    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"]
Nov 04 00:13:32.760128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.758482    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.853129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.851735    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.853129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.851852    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6bh\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.853129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.851887    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.853129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.852039    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.918793 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:32.916199    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d8c31b_2b80_4809_822f_077d637d7e18.slice/crio-a0f6af29016db1c017765659c149f5b047e6e3ea8ed730707a267bb34ed8c848 WatchSource:0}: Error finding container a0f6af29016db1c017765659c149f5b047e6e3ea8ed730707a267bb34ed8c848: Status 404 returned error can't find the container with id a0f6af29016db1c017765659c149f5b047e6e3ea8ed730707a267bb34ed8c848
Nov 04 00:13:32.918793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.918078    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"]
Nov 04 00:13:32.953400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.953359    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.953616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.953595    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.953713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.953656    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6bh\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.953713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.953684    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.954299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.954276    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.954620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.954598    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.956421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.956401    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:32.967895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:32.967865    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6bh\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh\") pod \"payload-processing-external-model-56f897f846-z8zxj\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:33.143701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:33.143614    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:33.423839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:33.423757    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb" event={"ID":"83d8c31b-2b80-4809-822f-077d637d7e18","Type":"ContainerStarted","Data":"64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e"}
Nov 04 00:13:33.423839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:33.423809    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb" event={"ID":"83d8c31b-2b80-4809-822f-077d637d7e18","Type":"ContainerStarted","Data":"a0f6af29016db1c017765659c149f5b047e6e3ea8ed730707a267bb34ed8c848"}
Nov 04 00:13:33.424091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:33.423881    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:33.433127 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:33.433092    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cea32ee_9fd3_4d0d_8b74_a85814b76840.slice/crio-a85d89972998d0d70c8892bc42abcb51ba4a6f925ad3bbe506d234b9e9cc77e9 WatchSource:0}: Error finding container a85d89972998d0d70c8892bc42abcb51ba4a6f925ad3bbe506d234b9e9cc77e9: Status 404 returned error can't find the container with id a85d89972998d0d70c8892bc42abcb51ba4a6f925ad3bbe506d234b9e9cc77e9
Nov 04 00:13:33.436400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:33.436372    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"]
Nov 04 00:13:33.456033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:33.455989    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb" podStartSLOduration=2.455976068 podStartE2EDuration="2.455976068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:33.447052326 +0000 UTC m=+2678.546652470" watchObservedRunningTime="2026-11-04 00:13:33.455976068 +0000 UTC m=+2678.555576230"
Nov 04 00:13:34.434383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:34.434341    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj" event={"ID":"8cea32ee-9fd3-4d0d-8b74-a85814b76840","Type":"ContainerStarted","Data":"d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e"}
Nov 04 00:13:34.434821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:34.434410    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj" event={"ID":"8cea32ee-9fd3-4d0d-8b74-a85814b76840","Type":"ContainerStarted","Data":"a85d89972998d0d70c8892bc42abcb51ba4a6f925ad3bbe506d234b9e9cc77e9"}
Nov 04 00:13:34.434821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:34.434439    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:34.460541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:34.460483    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj" podStartSLOduration=2.460469643 podStartE2EDuration="2.460469643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:34.458348323 +0000 UTC m=+2679.557948468" watchObservedRunningTime="2026-11-04 00:13:34.460469643 +0000 UTC m=+2679.560069788"
Nov 04 00:13:37.118844 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:37.118801    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a43cca408b12: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:37.118844 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:37.118846    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:37.119408 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:37.119188    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials podName:5e485bf1-7f02-4ed0-b5f0-f12fdd46be26 nodeName:}" failed. No retries permitted until 2026-11-04 00:13:37.619165959 +0000 UTC m=+2682.718766082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials") pod "payload-processing-external-model-6bd64bd569-vw4cq" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26") : secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:37.413986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.413875    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx" podUID="e0c0f7db-e15b-4cfc-894d-832cfdd87b61" containerName="payload-processing" containerID="cri-o://8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1" gracePeriod=25
Nov 04 00:13:37.627216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:37.626778    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a43cca408b12: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:37.627216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:37.626813    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:37.627216 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:37.626871    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials podName:5e485bf1-7f02-4ed0-b5f0-f12fdd46be26 nodeName:}" failed. No retries permitted until 2026-11-04 00:13:38.62685367 +0000 UTC m=+2683.726453810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials") pod "payload-processing-external-model-6bd64bd569-vw4cq" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26") : secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:37.715811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.715783    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:37.831118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831086    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume\") pod \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") "
Nov 04 00:13:37.831269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831217    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials\") pod \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") "
Nov 04 00:13:37.831269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831239    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume\") pod \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") "
Nov 04 00:13:37.831371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831343    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd\") pod \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\" (UID: \"e0c0f7db-e15b-4cfc-894d-832cfdd87b61\") "
Nov 04 00:13:37.831579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831555    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume" pod "e0c0f7db-e15b-4cfc-894d-832cfdd87b61" (UID: "e0c0f7db-e15b-4cfc-894d-832cfdd87b61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:37.831714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831690    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume" pod "e0c0f7db-e15b-4cfc-894d-832cfdd87b61" (UID: "e0c0f7db-e15b-4cfc-894d-832cfdd87b61"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:37.831965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831947    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:37.832014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.831972    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:37.833569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.833542    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd" pod "e0c0f7db-e15b-4cfc-894d-832cfdd87b61" (UID: "e0c0f7db-e15b-4cfc-894d-832cfdd87b61"). InnerVolumeSpecName "kube-api-access-9ffbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:37.835674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.835653    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials" pod "e0c0f7db-e15b-4cfc-894d-832cfdd87b61" (UID: "e0c0f7db-e15b-4cfc-894d-832cfdd87b61"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:37.933533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.933505    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ffbd\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-kube-api-access-9ffbd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:37.933533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:37.933533    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c0f7db-e15b-4cfc-894d-832cfdd87b61-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:38.424675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.424578    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb" podUID="83d8c31b-2b80-4809-822f-077d637d7e18" containerName="payload-processing" containerID="cri-o://64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e" gracePeriod=25
Nov 04 00:13:38.474093 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.474062    5074 generic.go:356] "Generic (PLEG): container finished" podID="e0c0f7db-e15b-4cfc-894d-832cfdd87b61" containerID="8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1" exitCode=0
Nov 04 00:13:38.474209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.474104    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"
Nov 04 00:13:38.474209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.474116    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx" event={"ID":"e0c0f7db-e15b-4cfc-894d-832cfdd87b61","Type":"ContainerDied","Data":"8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1"}
Nov 04 00:13:38.474209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.474157    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-rzrdx" event={"ID":"e0c0f7db-e15b-4cfc-894d-832cfdd87b61","Type":"ContainerDied","Data":"ffd57e699792b51b7f748a9199b0b5ee0c1d57fb0e755ef9bf7c073c60b17ec3"}
Nov 04 00:13:38.474209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.474177    5074 scope.go:122] "RemoveContainer" containerID="8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1"
Nov 04 00:13:38.540002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.539972    5074 scope.go:122] "RemoveContainer" containerID="8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1"
Nov 04 00:13:38.540885 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:38.540854    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1\": container with ID starting with 8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1 not found: ID does not exist" containerID="8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1"
Nov 04 00:13:38.541000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.540895    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1"} err="failed to get container status \"8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1\": rpc error: code = NotFound desc = could not find container \"8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1\": container with ID starting with 8f72ca4f916e18adb99332a5d4a5c777eb3bc5e57dadf28d995726171ee00fc1 not found: ID does not exist"
Nov 04 00:13:38.542850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.542824    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"]
Nov 04 00:13:38.549067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.549040    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-rzrdx"]
Nov 04 00:13:38.642239 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:38.642209    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a43cca408b12: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:38.642239 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:38.642242    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:38.642484 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:38.642307    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials podName:5e485bf1-7f02-4ed0-b5f0-f12fdd46be26 nodeName:}" failed. No retries permitted until 2026-11-04 00:13:40.642288225 +0000 UTC m=+2685.741888348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials") pod "payload-processing-external-model-6bd64bd569-vw4cq" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26") : secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:38.718293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.718259    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:38.845082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845041    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2\") pod \"83d8c31b-2b80-4809-822f-077d637d7e18\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") "
Nov 04 00:13:38.845269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845095    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume\") pod \"83d8c31b-2b80-4809-822f-077d637d7e18\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") "
Nov 04 00:13:38.845269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845149    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume\") pod \"83d8c31b-2b80-4809-822f-077d637d7e18\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") "
Nov 04 00:13:38.845269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845257    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials\") pod \"83d8c31b-2b80-4809-822f-077d637d7e18\" (UID: \"83d8c31b-2b80-4809-822f-077d637d7e18\") "
Nov 04 00:13:38.845571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845542    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume" pod "83d8c31b-2b80-4809-822f-077d637d7e18" (UID: "83d8c31b-2b80-4809-822f-077d637d7e18"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:38.845654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845615    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume" pod "83d8c31b-2b80-4809-822f-077d637d7e18" (UID: "83d8c31b-2b80-4809-822f-077d637d7e18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:38.846011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.845984    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:38.846071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.846016    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d8c31b-2b80-4809-822f-077d637d7e18-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:38.847898 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.847873    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials" pod "83d8c31b-2b80-4809-822f-077d637d7e18" (UID: "83d8c31b-2b80-4809-822f-077d637d7e18"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:38.847996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.847905    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2" pod "83d8c31b-2b80-4809-822f-077d637d7e18" (UID: "83d8c31b-2b80-4809-822f-077d637d7e18"). InnerVolumeSpecName "kube-api-access-kxnm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:38.947024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.946988    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxnm2\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-kube-api-access-kxnm2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:38.947024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:38.947021    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83d8c31b-2b80-4809-822f-077d637d7e18-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:39.026255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.026208    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c0f7db-e15b-4cfc-894d-832cfdd87b61" path="/var/lib/kubelet/pods/e0c0f7db-e15b-4cfc-894d-832cfdd87b61/volumes"
Nov 04 00:13:39.485170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.485134    5074 generic.go:356] "Generic (PLEG): container finished" podID="83d8c31b-2b80-4809-822f-077d637d7e18" containerID="64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e" exitCode=0
Nov 04 00:13:39.485615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.485177    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"
Nov 04 00:13:39.485615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.485221    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb" event={"ID":"83d8c31b-2b80-4809-822f-077d637d7e18","Type":"ContainerDied","Data":"64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e"}
Nov 04 00:13:39.485615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.485273    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb" event={"ID":"83d8c31b-2b80-4809-822f-077d637d7e18","Type":"ContainerDied","Data":"a0f6af29016db1c017765659c149f5b047e6e3ea8ed730707a267bb34ed8c848"}
Nov 04 00:13:39.485615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.485296    5074 scope.go:122] "RemoveContainer" containerID="64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e"
Nov 04 00:13:39.496300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.496280    5074 scope.go:122] "RemoveContainer" containerID="64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e"
Nov 04 00:13:39.496583 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:39.496561    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e\": container with ID starting with 64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e not found: ID does not exist" containerID="64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e"
Nov 04 00:13:39.496692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.496592    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e"} err="failed to get container status \"64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e\": rpc error: code = NotFound desc = could not find container \"64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e\": container with ID starting with 64c9a17293c3bbdfa3be7498e594aa2656802efee22c9a781f0f149a5ba1e33e not found: ID does not exist"
Nov 04 00:13:39.510073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.510050    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"]
Nov 04 00:13:39.515643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:39.515623    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-6qrfb"]
Nov 04 00:13:40.670046 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:40.670014    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a43cca408b12: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:40.670046 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:40.670046    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:40.670477 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:40.670113    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials podName:5e485bf1-7f02-4ed0-b5f0-f12fdd46be26 nodeName:}" failed. No retries permitted until 2026-11-04 00:13:44.670098497 +0000 UTC m=+2689.769698640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials") pod "payload-processing-external-model-6bd64bd569-vw4cq" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26") : secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:41.027652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:41.027618    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d8c31b-2b80-4809-822f-077d637d7e18" path="/var/lib/kubelet/pods/83d8c31b-2b80-4809-822f-077d637d7e18/volumes"
Nov 04 00:13:44.173429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.173391    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"]
Nov 04 00:13:44.176802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.176776    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:44.209532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.209500    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"]
Nov 04 00:13:44.210412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.210391    5074 cpu_manager.go:414] "containerMap: removing container" podUID="83d8c31b-2b80-4809-822f-077d637d7e18" containerName="payload-processing"
Nov 04 00:13:44.210412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.210412    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d8c31b-2b80-4809-822f-077d637d7e18" containerName="payload-processing"
Nov 04 00:13:44.210550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.210455    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e0c0f7db-e15b-4cfc-894d-832cfdd87b61" containerName="payload-processing"
Nov 04 00:13:44.210550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.210460    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c0f7db-e15b-4cfc-894d-832cfdd87b61" containerName="payload-processing"
Nov 04 00:13:44.210618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.210573    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c0f7db-e15b-4cfc-894d-832cfdd87b61" containerName="payload-processing"
Nov 04 00:13:44.210618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.210585    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d8c31b-2b80-4809-822f-077d637d7e18" containerName="payload-processing"
Nov 04 00:13:44.216589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.216549    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.220980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.220909    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"]
Nov 04 00:13:44.223114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.223091    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f0ff239bd626\""
Nov 04 00:13:44.328274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.328241    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbh9\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.328537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.328308    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.328537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.328409    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.328537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.328456    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.438753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.434357    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.438753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.434512    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.438753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.434592    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.438753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.434927    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbh9\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.438753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.435815    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.438753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.436186    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.451059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.451030    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbh9\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.451059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.451053    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials\") pod \"payload-processing-external-model-9d7d778bd-x7w2z\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") " pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.539094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.539055    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:44.749073 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:44.748993    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a43cca408b12: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:44.749296 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:44.749282    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:44.749428 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:44.749419    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials podName:5e485bf1-7f02-4ed0-b5f0-f12fdd46be26 nodeName:}" failed. No retries permitted until 2026-11-04 00:13:52.749399273 +0000 UTC m=+2697.848999398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials") pod "payload-processing-external-model-6bd64bd569-vw4cq" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26") : secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:44.850519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.850290    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"]
Nov 04 00:13:44.953759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.942181    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"]
Nov 04 00:13:44.991925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:44.990976    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-xm45v"]
Nov 04 00:13:45.084016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.083841    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.105820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.105791    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-xm45v"]
Nov 04 00:13:45.158842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.158806    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8d46\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.159040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.158871    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.159124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.159056    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.159124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.159099    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.260397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.260361    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.261013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.260425    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.261013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.260703    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8d46\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.261013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.260809    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.261603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.261576    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.261781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.261673    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.263716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.263690    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.273793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.273761    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8d46\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46\") pod \"payload-processing-external-model-56f897f846-xm45v\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") " pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.417122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.417025    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:45.564850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.564573    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z" event={"ID":"db8e64d6-f357-43bd-afa4-cd022445c2fb","Type":"ContainerStarted","Data":"369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d"}
Nov 04 00:13:45.565030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.564881    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:45.565030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.564915    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z" event={"ID":"db8e64d6-f357-43bd-afa4-cd022445c2fb","Type":"ContainerStarted","Data":"9fa1aaf1511de820b92ee4146161ca16ac21207a633e0371c265e2c34ca2596b"}
Nov 04 00:13:45.589726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.589668    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z" podStartSLOduration=1.5896537130000001 podStartE2EDuration="1.589653713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:45.586932448 +0000 UTC m=+2690.686532618" watchObservedRunningTime="2026-11-04 00:13:45.589653713 +0000 UTC m=+2690.689253854"
Nov 04 00:13:45.743450 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:45.743419    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09c3065_5cd0_407b_9adb_8fb78e8047cb.slice/crio-9c4f402ee53848c77c6f89f006785f1e281f3f4a3b9d4cfb2d364eb00b7ddd2a WatchSource:0}: Error finding container 9c4f402ee53848c77c6f89f006785f1e281f3f4a3b9d4cfb2d364eb00b7ddd2a: Status 404 returned error can't find the container with id 9c4f402ee53848c77c6f89f006785f1e281f3f4a3b9d4cfb2d364eb00b7ddd2a
Nov 04 00:13:45.743889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:45.743862    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-xm45v"]
Nov 04 00:13:46.574353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:46.574315    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v" event={"ID":"d09c3065-5cd0-407b-9adb-8fb78e8047cb","Type":"ContainerStarted","Data":"c559b14d85d66339de8bd739767a35c07a1035d6c7a2e1c9a1c3d289afbf2f3c"}
Nov 04 00:13:46.574781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:46.574373    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v" event={"ID":"d09c3065-5cd0-407b-9adb-8fb78e8047cb","Type":"ContainerStarted","Data":"9c4f402ee53848c77c6f89f006785f1e281f3f4a3b9d4cfb2d364eb00b7ddd2a"}
Nov 04 00:13:46.574781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:46.574406    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:46.606998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:46.606937    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v" podStartSLOduration=2.60692052 podStartE2EDuration="2.60692052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:46.598590512 +0000 UTC m=+2691.698190657" watchObservedRunningTime="2026-11-04 00:13:46.60692052 +0000 UTC m=+2691.706520664"
Nov 04 00:13:49.173843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.173778    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj" podUID="8cea32ee-9fd3-4d0d-8b74-a85814b76840" containerName="payload-processing" containerID="cri-o://d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e" gracePeriod=25
Nov 04 00:13:49.511689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.511656    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:49.600645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.600610    5074 generic.go:356] "Generic (PLEG): container finished" podID="8cea32ee-9fd3-4d0d-8b74-a85814b76840" containerID="d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e" exitCode=0
Nov 04 00:13:49.600813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.600653    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"
Nov 04 00:13:49.600813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.600697    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj" event={"ID":"8cea32ee-9fd3-4d0d-8b74-a85814b76840","Type":"ContainerDied","Data":"d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e"}
Nov 04 00:13:49.600813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.600773    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-z8zxj" event={"ID":"8cea32ee-9fd3-4d0d-8b74-a85814b76840","Type":"ContainerDied","Data":"a85d89972998d0d70c8892bc42abcb51ba4a6f925ad3bbe506d234b9e9cc77e9"}
Nov 04 00:13:49.600813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.600804    5074 scope.go:122] "RemoveContainer" containerID="d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e"
Nov 04 00:13:49.614020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.613992    5074 scope.go:122] "RemoveContainer" containerID="d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e"
Nov 04 00:13:49.614339 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:49.614318    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e\": container with ID starting with d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e not found: ID does not exist" containerID="d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e"
Nov 04 00:13:49.614425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.614351    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e"} err="failed to get container status \"d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e\": rpc error: code = NotFound desc = could not find container \"d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e\": container with ID starting with d71996e164a9dd1d603773accb42a17e01b8cc35d44211f2f66de5950c510b0e not found: ID does not exist"
Nov 04 00:13:49.632011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.631984    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume\") pod \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") "
Nov 04 00:13:49.632213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632197    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh\") pod \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") "
Nov 04 00:13:49.632348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632334    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume\") pod \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") "
Nov 04 00:13:49.632428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632373    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials\") pod \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\" (UID: \"8cea32ee-9fd3-4d0d-8b74-a85814b76840\") "
Nov 04 00:13:49.632428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632383    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume" pod "8cea32ee-9fd3-4d0d-8b74-a85814b76840" (UID: "8cea32ee-9fd3-4d0d-8b74-a85814b76840"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:49.632823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632796    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume" pod "8cea32ee-9fd3-4d0d-8b74-a85814b76840" (UID: "8cea32ee-9fd3-4d0d-8b74-a85814b76840"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:49.632979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632959    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:49.633040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.632985    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cea32ee-9fd3-4d0d-8b74-a85814b76840-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:49.635056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.634516    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh" pod "8cea32ee-9fd3-4d0d-8b74-a85814b76840" (UID: "8cea32ee-9fd3-4d0d-8b74-a85814b76840"). InnerVolumeSpecName "kube-api-access-gs6bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:49.635056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.634767    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials" pod "8cea32ee-9fd3-4d0d-8b74-a85814b76840" (UID: "8cea32ee-9fd3-4d0d-8b74-a85814b76840"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:49.754813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.745919    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6bh\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-kube-api-access-gs6bh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:49.754813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.745958    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cea32ee-9fd3-4d0d-8b74-a85814b76840-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:49.940482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.939814    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"]
Nov 04 00:13:49.947039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:49.946126    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-z8zxj"]
Nov 04 00:13:50.565408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:50.565352    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z" podUID="db8e64d6-f357-43bd-afa4-cd022445c2fb" containerName="payload-processing" containerID="cri-o://369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d" gracePeriod=25
Nov 04 00:13:51.172769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.172074    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:51.301810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.275715    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cea32ee-9fd3-4d0d-8b74-a85814b76840" path="/var/lib/kubelet/pods/8cea32ee-9fd3-4d0d-8b74-a85814b76840/volumes"
Nov 04 00:13:51.301810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.282944    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9\") pod \"db8e64d6-f357-43bd-afa4-cd022445c2fb\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") "
Nov 04 00:13:51.334526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.329896    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume\") pod \"db8e64d6-f357-43bd-afa4-cd022445c2fb\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") "
Nov 04 00:13:51.340116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.340027    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume" pod "db8e64d6-f357-43bd-afa4-cd022445c2fb" (UID: "db8e64d6-f357-43bd-afa4-cd022445c2fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:51.340583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.340546    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials\") pod \"db8e64d6-f357-43bd-afa4-cd022445c2fb\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") "
Nov 04 00:13:51.341244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.341141    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume\") pod \"db8e64d6-f357-43bd-afa4-cd022445c2fb\" (UID: \"db8e64d6-f357-43bd-afa4-cd022445c2fb\") "
Nov 04 00:13:51.352778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.348513    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:51.403777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.394912    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9" pod "db8e64d6-f357-43bd-afa4-cd022445c2fb" (UID: "db8e64d6-f357-43bd-afa4-cd022445c2fb"). InnerVolumeSpecName "kube-api-access-rbbh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:51.403777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.397583    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume" pod "db8e64d6-f357-43bd-afa4-cd022445c2fb" (UID: "db8e64d6-f357-43bd-afa4-cd022445c2fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:13:51.445811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.441125    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials" pod "db8e64d6-f357-43bd-afa4-cd022445c2fb" (UID: "db8e64d6-f357-43bd-afa4-cd022445c2fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:13:51.471871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.466151    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbh9\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-kube-api-access-rbbh9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:51.471871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.466188    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db8e64d6-f357-43bd-afa4-cd022445c2fb-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:51.471871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.466208    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db8e64d6-f357-43bd-afa4-cd022445c2fb-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:13:51.774763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.774594    5074 generic.go:356] "Generic (PLEG): container finished" podID="db8e64d6-f357-43bd-afa4-cd022445c2fb" containerID="369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d" exitCode=0
Nov 04 00:13:51.775299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.775113    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z" event={"ID":"db8e64d6-f357-43bd-afa4-cd022445c2fb","Type":"ContainerDied","Data":"369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d"}
Nov 04 00:13:51.775299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.775249    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z" event={"ID":"db8e64d6-f357-43bd-afa4-cd022445c2fb","Type":"ContainerDied","Data":"9fa1aaf1511de820b92ee4146161ca16ac21207a633e0371c265e2c34ca2596b"}
Nov 04 00:13:51.775444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.775324    5074 scope.go:122] "RemoveContainer" containerID="369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d"
Nov 04 00:13:51.798797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:51.775595    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"
Nov 04 00:13:52.090791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.078614    5074 scope.go:122] "RemoveContainer" containerID="369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d"
Nov 04 00:13:52.090791 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:52.079681    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d\": container with ID starting with 369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d not found: ID does not exist" containerID="369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d"
Nov 04 00:13:52.090791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.079725    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d"} err="failed to get container status \"369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d\": rpc error: code = NotFound desc = could not find container \"369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d\": container with ID starting with 369a46ae80f0f7471472d64dc16f604d7a79d3d61bcd3aede2d6917350a4613d not found: ID does not exist"
Nov 04 00:13:52.151875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.150792    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"]
Nov 04 00:13:52.238765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.235205    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d7d778bd-x7w2z"]
Nov 04 00:13:52.426820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.411928    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 04 00:13:52.450774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.450047    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 04 00:13:52.723959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.722819    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 04 00:13:52.749489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:52.748814    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 04 00:13:52.816868 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:52.816401    5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a43cca408b12: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:52.816868 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:52.816440    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq: secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:52.816868 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:52.816513    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials podName:5e485bf1-7f02-4ed0-b5f0-f12fdd46be26 nodeName:}" failed. No retries permitted until 2026-11-04 00:14:08.816494172 +0000 UTC m=+2713.916094307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials") pod "payload-processing-external-model-6bd64bd569-vw4cq" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26") : secret "agc-cred-a43cca408b12" not found
Nov 04 00:13:53.089814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:53.079087    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8e64d6-f357-43bd-afa4-cd022445c2fb" path="/var/lib/kubelet/pods/db8e64d6-f357-43bd-afa4-cd022445c2fb/volumes"
Nov 04 00:13:54.289390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.289353    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"]
Nov 04 00:13:54.290409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.290383    5074 cpu_manager.go:414] "containerMap: removing container" podUID="db8e64d6-f357-43bd-afa4-cd022445c2fb" containerName="payload-processing"
Nov 04 00:13:54.290409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.290409    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8e64d6-f357-43bd-afa4-cd022445c2fb" containerName="payload-processing"
Nov 04 00:13:54.290612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.290451    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8cea32ee-9fd3-4d0d-8b74-a85814b76840" containerName="payload-processing"
Nov 04 00:13:54.290612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.290460    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cea32ee-9fd3-4d0d-8b74-a85814b76840" containerName="payload-processing"
Nov 04 00:13:54.290723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.290711    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8e64d6-f357-43bd-afa4-cd022445c2fb" containerName="payload-processing"
Nov 04 00:13:54.290804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.290784    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cea32ee-9fd3-4d0d-8b74-a85814b76840" containerName="payload-processing"
Nov 04 00:13:54.299116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.299089    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.302181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.302150    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"]
Nov 04 00:13:54.311818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.311790    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"openshift-service-ca.crt\""
Nov 04 00:13:54.312513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.312223    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"routing-overlay\""
Nov 04 00:13:54.312668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.312292    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"payload-processing-external-model-e2e--36d07d4b-dockercfg-6x5ks\""
Nov 04 00:13:54.312784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.312322    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"agc-cred-545f9f70b029\""
Nov 04 00:13:54.312941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.312396    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"kube-root-ca.crt\""
Nov 04 00:13:54.313060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.312460    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c7d218\""
Nov 04 00:13:54.481516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.481468    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.483470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.483441    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.483790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.483728    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.484144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.484124    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4fpk\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.586379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.586275    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.586379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.586375    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.586622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.586409    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fpk\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.586689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.586666    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.587223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.587185    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.587598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.587576    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.589104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.589081    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.600378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.600351    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fpk\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.633772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.633185    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:54.931636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:54.931599    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"]
Nov 04 00:13:54.931792 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:54.931715    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ee1001_32b6_46bc_9329_9a49589e28e7.slice/crio-1302eafce3006ab9016c518fbd260cc1cc5bea8673a72f30c5e0b8caf3292625 WatchSource:0}: Error finding container 1302eafce3006ab9016c518fbd260cc1cc5bea8673a72f30c5e0b8caf3292625: Status 404 returned error can't find the container with id 1302eafce3006ab9016c518fbd260cc1cc5bea8673a72f30c5e0b8caf3292625
Nov 04 00:13:55.750271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:55.750238    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:13:55.754302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:55.754279    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:13:55.893232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:55.893190    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th" event={"ID":"e4ee1001-32b6-46bc-9329-9a49589e28e7","Type":"ContainerStarted","Data":"00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f"}
Nov 04 00:13:55.893410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:55.893296    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:13:55.893410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:55.893323    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th" event={"ID":"e4ee1001-32b6-46bc-9329-9a49589e28e7","Type":"ContainerStarted","Data":"1302eafce3006ab9016c518fbd260cc1cc5bea8673a72f30c5e0b8caf3292625"}
Nov 04 00:13:55.919331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:55.919255    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th" podStartSLOduration=1.919231965 podStartE2EDuration="1.919231965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:55.91283798 +0000 UTC m=+2701.012438125" watchObservedRunningTime="2026-11-04 00:13:55.919231965 +0000 UTC m=+2701.018832110"
Nov 04 00:13:57.601974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:57.601936    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:13:57.789852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:57.779515    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"]
Nov 04 00:13:58.123904 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:13:58.120928    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:13:58.473859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.473826    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"]
Nov 04 00:13:58.478804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.478787    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.483860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.483835    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a002b\""
Nov 04 00:13:58.484016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.483835    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"openshift-service-ca.crt\""
Nov 04 00:13:58.484016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.483877    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"routing-overlay\""
Nov 04 00:13:58.484143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.483920    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"kube-root-ca.crt\""
Nov 04 00:13:58.490993 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.490964    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"]
Nov 04 00:13:58.576475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.576441    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"agc-cred-14b358f9ee72\""
Nov 04 00:13:58.576725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.576705    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-e2e--7c418871-dockercfg-7sffv\""
Nov 04 00:13:58.578421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.578398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.578485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.578465    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.578622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.578605    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.578672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.578646    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqkh\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.695822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.681653    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.695822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.681805    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.695822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.681965    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.695822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.682029    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqkh\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.695822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.684526    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.695822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.691891    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.778401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.778288    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.779252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.779224    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqkh\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.807811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.807532    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:58.875705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.875664    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"]
Nov 04 00:13:58.885049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.885016    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:58.892826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.891827    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"]
Nov 04 00:13:58.894708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.894644    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\""
Nov 04 00:13:58.894842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.894796    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\""
Nov 04 00:13:58.894920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.894902    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\""
Nov 04 00:13:58.895787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.895108    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-d6htd\""
Nov 04 00:13:58.895787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.895177    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-559c901bbcfd\""
Nov 04 00:13:58.895787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.895261    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\""
Nov 04 00:13:58.992895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.992848    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:58.993069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.992984    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:58.993139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.993065    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:58.993231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:58.993212    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkn2t\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.113998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.098301    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.113998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.109834    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.113998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.109930    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.113998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.110104    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkn2t\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.113998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.110266    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.113998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.111175    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.145344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.145306    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.146822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.146759    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkn2t\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.189462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.189427    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"]
Nov 04 00:13:59.222118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.222086    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.494573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.494540    5074 scope.go:122] "RemoveContainer" containerID="ecee004215112eacbc87abe11e239bc6e12610ad021f4ff88f175e77c137d152"
Nov 04 00:13:59.508186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.508154    5074 scope.go:122] "RemoveContainer" containerID="5eda02dcb5c33d3a9a8d3e7d498c59761ba5ca0f85f0cb6ef4a502657d8e18f8"
Nov 04 00:13:59.521483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.521453    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"]
Nov 04 00:13:59.523126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.523081    5074 scope.go:122] "RemoveContainer" containerID="52cfd6570478c2a23506d5276ac61c8f43c28d0c48124c3aa39229e1bc161784"
Nov 04 00:13:59.525176 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:13:59.525145    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c940782_6686_4f19_bea5_2ab8dfd2e0f7.slice/crio-c5dc1aeaccd87f3c17b5cd64220beac33627a37562bf398a4316205b68322f3e WatchSource:0}: Error finding container c5dc1aeaccd87f3c17b5cd64220beac33627a37562bf398a4316205b68322f3e: Status 404 returned error can't find the container with id c5dc1aeaccd87f3c17b5cd64220beac33627a37562bf398a4316205b68322f3e
Nov 04 00:13:59.953213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.953154    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l" event={"ID":"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e","Type":"ContainerStarted","Data":"7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939"}
Nov 04 00:13:59.953767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.953239    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:13:59.953767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.953267    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l" event={"ID":"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e","Type":"ContainerStarted","Data":"f3d5fe85b11c3d6c63a87d6992cb9a1e1f2c85fb0d291a9c57f93f289f56c391"}
Nov 04 00:13:59.955070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.955032    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m" event={"ID":"0c940782-6686-4f19-bea5-2ab8dfd2e0f7","Type":"ContainerStarted","Data":"a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a"}
Nov 04 00:13:59.955199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.955093    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m" event={"ID":"0c940782-6686-4f19-bea5-2ab8dfd2e0f7","Type":"ContainerStarted","Data":"c5dc1aeaccd87f3c17b5cd64220beac33627a37562bf398a4316205b68322f3e"}
Nov 04 00:13:59.955199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.955126    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:13:59.978338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.978287    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l" podStartSLOduration=1.978273389 podStartE2EDuration="1.978273389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:59.973392625 +0000 UTC m=+2705.072992781" watchObservedRunningTime="2026-11-04 00:13:59.978273389 +0000 UTC m=+2705.077873533"
Nov 04 00:13:59.997536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:13:59.997480    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m" podStartSLOduration=1.99746485 podStartE2EDuration="1.99746485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:13:59.993436136 +0000 UTC m=+2705.093036282" watchObservedRunningTime="2026-11-04 00:13:59.99746485 +0000 UTC m=+2705.097064995"
Nov 04 00:14:02.780823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:02.780778    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq" podUID="5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" containerName="payload-processing" containerID="cri-o://724888e1b1b22ecd61f700be128b812986476096883a7ffbc504f140f740d12b" gracePeriod=25
Nov 04 00:14:02.984914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:02.982834    5074 generic.go:356] "Generic (PLEG): container finished" podID="5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" containerID="724888e1b1b22ecd61f700be128b812986476096883a7ffbc504f140f740d12b" exitCode=0
Nov 04 00:14:02.984914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:02.982908    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq" event={"ID":"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26","Type":"ContainerDied","Data":"724888e1b1b22ecd61f700be128b812986476096883a7ffbc504f140f740d12b"}
Nov 04 00:14:03.032150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.032061    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"]
Nov 04 00:14:03.036914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.036884    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db80dd92-8zrhn"]
Nov 04 00:14:03.096795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.096769    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:14:03.178240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.178205    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685\") pod \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") "
Nov 04 00:14:03.178409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.178392    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials\") pod \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") "
Nov 04 00:14:03.178482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.178448    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume\") pod \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") "
Nov 04 00:14:03.178545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.178500    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume\") pod \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\" (UID: \"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26\") "
Nov 04 00:14:03.178981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.178952    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume" pod "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:03.179106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.179023    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume" pod "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:03.179249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.179230    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:03.179314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.179254    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:03.181162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.181137    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685" pod "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26"). InnerVolumeSpecName "kube-api-access-md685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:03.181299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.181279    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials" pod "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" (UID: "5e485bf1-7f02-4ed0-b5f0-f12fdd46be26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:03.280952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.280919    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md685\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-kube-api-access-md685\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:03.280952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.280952    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:03.993777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.993732    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"
Nov 04 00:14:03.993777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.993769    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq" event={"ID":"5e485bf1-7f02-4ed0-b5f0-f12fdd46be26","Type":"ContainerDied","Data":"4fb9c8120f2283d0951ee11df4f461676928a7d1e3f5954eb4519b0680cc8950"}
Nov 04 00:14:03.994256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:03.993837    5074 scope.go:122] "RemoveContainer" containerID="724888e1b1b22ecd61f700be128b812986476096883a7ffbc504f140f740d12b"
Nov 04 00:14:04.023877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:04.023846    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"]
Nov 04 00:14:04.028219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:04.028196    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd64bd569-vw4cq"]
Nov 04 00:14:05.020594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:05.020556    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" path="/var/lib/kubelet/pods/5e485bf1-7f02-4ed0-b5f0-f12fdd46be26/volumes"
Nov 04 00:14:05.021139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:05.021079    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef7b6cc-8bae-498b-b803-70bb9d931ca6" path="/var/lib/kubelet/pods/eef7b6cc-8bae-498b-b803-70bb9d931ca6/volumes"
Nov 04 00:14:06.907619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:06.907591    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:14:09.471692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.471655    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"]
Nov 04 00:14:09.472501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.472483    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" containerName="payload-processing"
Nov 04 00:14:09.472553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.472502    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" containerName="payload-processing"
Nov 04 00:14:09.472642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.472633    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e485bf1-7f02-4ed0-b5f0-f12fdd46be26" containerName="payload-processing"
Nov 04 00:14:09.478229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.478205    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.485142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.484378    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"]
Nov 04 00:14:09.485564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.485500    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-241087\"/\"kube-root-ca.crt\""
Nov 04 00:14:09.485942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.485917    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-241087\"/\"default-dockercfg-f7vht\""
Nov 04 00:14:09.486652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.486625    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-241087\"/\"openshift-service-ca.crt\""
Nov 04 00:14:09.487285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.487257    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-241087\"/\"e2e-usage-sim-10ac07-cert\""
Nov 04 00:14:09.572961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.572922    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwr45\" (UniqueName: \"kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45\") pod \"e2e-usage-sim-10ac07-6c4b9c78f-58clq\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") " pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.572961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.572971    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls\") pod \"e2e-usage-sim-10ac07-6c4b9c78f-58clq\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") " pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.674170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.674134    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwr45\" (UniqueName: \"kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45\") pod \"e2e-usage-sim-10ac07-6c4b9c78f-58clq\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") " pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.674351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.674183    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls\") pod \"e2e-usage-sim-10ac07-6c4b9c78f-58clq\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") " pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.676711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.676683    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls\") pod \"e2e-usage-sim-10ac07-6c4b9c78f-58clq\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") " pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.687256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.687227    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwr45\" (UniqueName: \"kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45\") pod \"e2e-usage-sim-10ac07-6c4b9c78f-58clq\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") " pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:09.823334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:09.823242    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:10.112602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:10.112558    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"]
Nov 04 00:14:10.113115 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:14:10.113086    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e15edf7_bded_4214_9a4d_097e9f44b4ed.slice/crio-2a819afed4ee52815ac763477443d56acb933838c853466fbd0f00e0b787bded WatchSource:0}: Error finding container 2a819afed4ee52815ac763477443d56acb933838c853466fbd0f00e0b787bded: Status 404 returned error can't find the container with id 2a819afed4ee52815ac763477443d56acb933838c853466fbd0f00e0b787bded
Nov 04 00:14:10.965849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:10.965814    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:14:10.966303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:10.965903    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:14:11.062395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:11.062354    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" event={"ID":"9e15edf7-bded-4214-9a4d-097e9f44b4ed","Type":"ContainerStarted","Data":"bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45"}
Nov 04 00:14:11.062595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:11.062403    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" event={"ID":"9e15edf7-bded-4214-9a4d-097e9f44b4ed","Type":"ContainerStarted","Data":"2a819afed4ee52815ac763477443d56acb933838c853466fbd0f00e0b787bded"}
Nov 04 00:14:11.062595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:11.062538    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:11.068003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:11.067976    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:14:11.083792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:11.083719    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" podStartSLOduration=2.08370477 podStartE2EDuration="2.08370477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:14:11.080636363 +0000 UTC m=+2716.180236507" watchObservedRunningTime="2026-11-04 00:14:11.08370477 +0000 UTC m=+2716.183304913"
Nov 04 00:14:15.439847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.439797    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"]
Nov 04 00:14:15.447787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.447757    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.456550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.456525    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 04 00:14:15.456912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.456889    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 04 00:14:15.457108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.457092    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 04 00:14:15.457885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.457864    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q44ht\""
Nov 04 00:14:15.458026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.457905    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"]
Nov 04 00:14:15.560836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.560799    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4wh\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.561023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.560850    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.561023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.560886    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.561156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.561119    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.561267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.561239    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.561337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.561313    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.561390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.561341    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.665322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.665269    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4wh\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.666297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.665589    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.666532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.666496    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.666780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.666762    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.666984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.666969    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.667233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.667209    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.668659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.668627    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.684802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.682926    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.699815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.698646    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.715730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.715669    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.721936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.721158    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.722486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.722465    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.723601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.723560    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.726299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.726245    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4wh\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pfdb4\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:15.779215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:15.778928    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:16.155795 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:14:16.155759    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73e0d3d_6097_429e_b198_35447cfa028e.slice/crio-3755431fef49de730e0bed1e6eb544d86c2d03478960ff0effa949d49b482b32 WatchSource:0}: Error finding container 3755431fef49de730e0bed1e6eb544d86c2d03478960ff0effa949d49b482b32: Status 404 returned error can't find the container with id 3755431fef49de730e0bed1e6eb544d86c2d03478960ff0effa949d49b482b32
Nov 04 00:14:16.156751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:16.156707    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"]
Nov 04 00:14:16.158958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:16.158924    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:14:16.159046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:16.158987    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:14:17.147765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:17.147362    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" event={"ID":"f73e0d3d-6097-429e-b198-35447cfa028e","Type":"ContainerStarted","Data":"acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f"}
Nov 04 00:14:17.147765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:17.147464    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" event={"ID":"f73e0d3d-6097-429e-b198-35447cfa028e","Type":"ContainerStarted","Data":"3755431fef49de730e0bed1e6eb544d86c2d03478960ff0effa949d49b482b32"}
Nov 04 00:14:17.148427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:17.147872    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:17.150186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:17.150154    5074 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-pfdb4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:14:17.150346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:17.150212    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" podUID="f73e0d3d-6097-429e-b198-35447cfa028e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:14:17.210905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:17.209724    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" podStartSLOduration=2.209705762 podStartE2EDuration="2.209705762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:14:17.204274116 +0000 UTC m=+2722.303874268" watchObservedRunningTime="2026-11-04 00:14:17.209705762 +0000 UTC m=+2722.309305907"
Nov 04 00:14:18.148877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:18.148845    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:14:21.038592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:21.038558    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"]
Nov 04 00:14:21.044227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:21.044194    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5daa7d31-mvbvz"]
Nov 04 00:14:23.025027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:23.024988    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e803824d-f79b-40d7-8e8e-1f7da989149f" path="/var/lib/kubelet/pods/e803824d-f79b-40d7-8e8e-1f7da989149f/volumes"
Nov 04 00:14:28.023522 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:14:28.023484    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:14:28.477497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:28.477446    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"]
Nov 04 00:14:28.492551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:28.492511    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"]
Nov 04 00:14:33.478836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.478769    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m" podUID="0c940782-6686-4f19-bea5-2ab8dfd2e0f7" containerName="payload-processing" containerID="cri-o://a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a" gracePeriod=25
Nov 04 00:14:33.493124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.493046    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l" podUID="92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" containerName="payload-processing" containerID="cri-o://7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939" gracePeriod=25
Nov 04 00:14:33.796416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.796386    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:14:33.872462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.872428    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:14:33.883277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.883253    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume\") pod \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") "
Nov 04 00:14:33.883407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.883291    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t\") pod \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") "
Nov 04 00:14:33.883407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.883338    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume\") pod \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") "
Nov 04 00:14:33.883560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.883416    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials\") pod \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\" (UID: \"0c940782-6686-4f19-bea5-2ab8dfd2e0f7\") "
Nov 04 00:14:33.883848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.883818    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume" pod "0c940782-6686-4f19-bea5-2ab8dfd2e0f7" (UID: "0c940782-6686-4f19-bea5-2ab8dfd2e0f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:33.883960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.883853    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume" pod "0c940782-6686-4f19-bea5-2ab8dfd2e0f7" (UID: "0c940782-6686-4f19-bea5-2ab8dfd2e0f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:33.884207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.884186    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:33.884207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.884207    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:33.885580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.885556    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials" pod "0c940782-6686-4f19-bea5-2ab8dfd2e0f7" (UID: "0c940782-6686-4f19-bea5-2ab8dfd2e0f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:33.885675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.885583    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t" pod "0c940782-6686-4f19-bea5-2ab8dfd2e0f7" (UID: "0c940782-6686-4f19-bea5-2ab8dfd2e0f7"). InnerVolumeSpecName "kube-api-access-zkn2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:33.984794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.984724    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume\") pod \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") "
Nov 04 00:14:33.984976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.984867    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume\") pod \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") "
Nov 04 00:14:33.985079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985003    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials\") pod \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") "
Nov 04 00:14:33.985162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985089    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh\") pod \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\" (UID: \"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e\") "
Nov 04 00:14:33.985257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985217    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume" pod "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" (UID: "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:33.985322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985272    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume" pod "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" (UID: "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:33.985601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985583    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkn2t\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-kube-api-access-zkn2t\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:33.985667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985604    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c940782-6686-4f19-bea5-2ab8dfd2e0f7-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:33.985667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985614    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:33.985667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.985623    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:33.987184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.987154    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials" pod "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" (UID: "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:33.987247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:33.987203    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh" pod "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" (UID: "92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e"). InnerVolumeSpecName "kube-api-access-2sqkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:34.086923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.086836    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:34.086923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.086872    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqkh\" (UniqueName: \"kubernetes.io/projected/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e-kube-api-access-2sqkh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:34.382718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.382623    5074 generic.go:356] "Generic (PLEG): container finished" podID="92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" containerID="7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939" exitCode=0
Nov 04 00:14:34.382718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.382687    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"
Nov 04 00:14:34.382718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.382705    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l" event={"ID":"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e","Type":"ContainerDied","Data":"7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939"}
Nov 04 00:14:34.383041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.382795    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l" event={"ID":"92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e","Type":"ContainerDied","Data":"f3d5fe85b11c3d6c63a87d6992cb9a1e1f2c85fb0d291a9c57f93f289f56c391"}
Nov 04 00:14:34.383041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.382827    5074 scope.go:122] "RemoveContainer" containerID="7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939"
Nov 04 00:14:34.384655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.384631    5074 generic.go:356] "Generic (PLEG): container finished" podID="0c940782-6686-4f19-bea5-2ab8dfd2e0f7" containerID="a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a" exitCode=0
Nov 04 00:14:34.384784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.384682    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"
Nov 04 00:14:34.384784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.384715    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m" event={"ID":"0c940782-6686-4f19-bea5-2ab8dfd2e0f7","Type":"ContainerDied","Data":"a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a"}
Nov 04 00:14:34.384879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.384837    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m" event={"ID":"0c940782-6686-4f19-bea5-2ab8dfd2e0f7","Type":"ContainerDied","Data":"c5dc1aeaccd87f3c17b5cd64220beac33627a37562bf398a4316205b68322f3e"}
Nov 04 00:14:34.404421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.399524    5074 scope.go:122] "RemoveContainer" containerID="7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939"
Nov 04 00:14:34.404421 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:14:34.400055    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939\": container with ID starting with 7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939 not found: ID does not exist" containerID="7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939"
Nov 04 00:14:34.404421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.400102    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939"} err="failed to get container status \"7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939\": rpc error: code = NotFound desc = could not find container \"7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939\": container with ID starting with 7de1e5336b1b31f634b6e66955c9833bdb9f6abf083f6bd5063341fd28a40939 not found: ID does not exist"
Nov 04 00:14:34.404421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.400128    5074 scope.go:122] "RemoveContainer" containerID="a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a"
Nov 04 00:14:34.415395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.415275    5074 scope.go:122] "RemoveContainer" containerID="a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a"
Nov 04 00:14:34.416069 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:14:34.416040    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a\": container with ID starting with a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a not found: ID does not exist" containerID="a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a"
Nov 04 00:14:34.416212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.416083    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a"} err="failed to get container status \"a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a\": rpc error: code = NotFound desc = could not find container \"a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a\": container with ID starting with a77026150c65228b4d8703256ce8a1a4585e1db16709f9bdedb5274d0890df9a not found: ID does not exist"
Nov 04 00:14:34.423871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.423849    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"]
Nov 04 00:14:34.431109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.431084    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-85596zl2m"]
Nov 04 00:14:34.437876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.437845    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"]
Nov 04 00:14:34.442309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:34.442285    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-8444sfs4l"]
Nov 04 00:14:35.021111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:35.021076    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c940782-6686-4f19-bea5-2ab8dfd2e0f7" path="/var/lib/kubelet/pods/0c940782-6686-4f19-bea5-2ab8dfd2e0f7/volumes"
Nov 04 00:14:35.021591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:35.021551    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" path="/var/lib/kubelet/pods/92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e/volumes"
Nov 04 00:14:47.289906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:47.289850    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:14:47.412117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:47.412050    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:14:47.626839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:47.626706    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:14:47.696853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:47.696811    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:14:47.697121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:47.697097    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:14:54.126361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.126323    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"]
Nov 04 00:14:54.127325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.127305    5074 cpu_manager.go:414] "containerMap: removing container" podUID="92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" containerName="payload-processing"
Nov 04 00:14:54.127393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.127326    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" containerName="payload-processing"
Nov 04 00:14:54.127393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.127347    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0c940782-6686-4f19-bea5-2ab8dfd2e0f7" containerName="payload-processing"
Nov 04 00:14:54.127393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.127352    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c940782-6686-4f19-bea5-2ab8dfd2e0f7" containerName="payload-processing"
Nov 04 00:14:54.127496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.127467    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c9e44f-e3d2-41ec-bd2b-b4ef3ed9e93e" containerName="payload-processing"
Nov 04 00:14:54.127496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.127477    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c940782-6686-4f19-bea5-2ab8dfd2e0f7" containerName="payload-processing"
Nov 04 00:14:54.133066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.133034    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.138937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.138828    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:14:54.140549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.140506    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"]
Nov 04 00:14:54.141904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.141871    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:14:54.314785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.314733    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.314977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.314846    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.314977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.314935    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djphp\" (UniqueName: \"kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.417216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.417121    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.417216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.417179    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.417452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.417241    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djphp\" (UniqueName: \"kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.418025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.417980    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.419773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.419729    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.430267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.430244    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djphp\" (UniqueName: \"kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp\") pod \"maas-api-revoke-sub-2a120cec-pclcq\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.457788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.457766    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:54.751520 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:14:54.751482    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62091dd9_fd32_4fa1_afd5_7ba971f2a163.slice/crio-0b90c2ae9c88d7fd34b2fa01c0b70a911754210d99b66c19976c46ec25bd9765 WatchSource:0}: Error finding container 0b90c2ae9c88d7fd34b2fa01c0b70a911754210d99b66c19976c46ec25bd9765: Status 404 returned error can't find the container with id 0b90c2ae9c88d7fd34b2fa01c0b70a911754210d99b66c19976c46ec25bd9765
Nov 04 00:14:54.751642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:54.751523    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"]
Nov 04 00:14:55.734069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:55.734024    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq" event={"ID":"62091dd9-fd32-4fa1-afd5-7ba971f2a163","Type":"ContainerStarted","Data":"bfe7821f8a64b422aa1014557ad9df99e5e1011e5fc1ca04bf06c0475ab502fd"}
Nov 04 00:14:55.734580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:55.734103    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq" event={"ID":"62091dd9-fd32-4fa1-afd5-7ba971f2a163","Type":"ContainerStarted","Data":"0b90c2ae9c88d7fd34b2fa01c0b70a911754210d99b66c19976c46ec25bd9765"}
Nov 04 00:14:55.756308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:55.756259    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq" podStartSLOduration=1.756246261 podStartE2EDuration="1.756246261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:14:55.753008605 +0000 UTC m=+2760.852608750" watchObservedRunningTime="2026-11-04 00:14:55.756246261 +0000 UTC m=+2760.855846404"
Nov 04 00:14:56.744358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:56.744318    5074 generic.go:356] "Generic (PLEG): container finished" podID="62091dd9-fd32-4fa1-afd5-7ba971f2a163" containerID="bfe7821f8a64b422aa1014557ad9df99e5e1011e5fc1ca04bf06c0475ab502fd" exitCode=0
Nov 04 00:14:56.744809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:56.744414    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq" event={"ID":"62091dd9-fd32-4fa1-afd5-7ba971f2a163","Type":"ContainerDied","Data":"bfe7821f8a64b422aa1014557ad9df99e5e1011e5fc1ca04bf06c0475ab502fd"}
Nov 04 00:14:57.965703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.965672    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:57.978066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.978037    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca\") pod \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") "
Nov 04 00:14:57.978235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.978195    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca\") pod \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") "
Nov 04 00:14:57.978235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.978223    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp\" (UniqueName: \"kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp\") pod \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\" (UID: \"62091dd9-fd32-4fa1-afd5-7ba971f2a163\") "
Nov 04 00:14:57.978551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.978526    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca" pod "62091dd9-fd32-4fa1-afd5-7ba971f2a163" (UID: "62091dd9-fd32-4fa1-afd5-7ba971f2a163"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:57.978801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.978783    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:57.980300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.980269    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca" pod "62091dd9-fd32-4fa1-afd5-7ba971f2a163" (UID: "62091dd9-fd32-4fa1-afd5-7ba971f2a163"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:14:57.980386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:57.980302    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp" pod "62091dd9-fd32-4fa1-afd5-7ba971f2a163" (UID: "62091dd9-fd32-4fa1-afd5-7ba971f2a163"). InnerVolumeSpecName "kube-api-access-djphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:58.045606 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:14:58.045501    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:14:58.079695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:58.079655    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djphp\" (UniqueName: \"kubernetes.io/projected/62091dd9-fd32-4fa1-afd5-7ba971f2a163-kube-api-access-djphp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.079695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:58.079693    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62091dd9-fd32-4fa1-afd5-7ba971f2a163-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.766800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:58.766767    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq" event={"ID":"62091dd9-fd32-4fa1-afd5-7ba971f2a163","Type":"ContainerDied","Data":"0b90c2ae9c88d7fd34b2fa01c0b70a911754210d99b66c19976c46ec25bd9765"}
Nov 04 00:14:58.767003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:58.766810    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"
Nov 04 00:14:58.767003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:14:58.766820    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b90c2ae9c88d7fd34b2fa01c0b70a911754210d99b66c19976c46ec25bd9765"
Nov 04 00:15:00.137680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.137644    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"]
Nov 04 00:15:00.138753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.138715    5074 cpu_manager.go:414] "containerMap: removing container" podUID="62091dd9-fd32-4fa1-afd5-7ba971f2a163" containerName="revoke-keys"
Nov 04 00:15:00.138876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.138758    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62091dd9-fd32-4fa1-afd5-7ba971f2a163" containerName="revoke-keys"
Nov 04 00:15:00.139003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.138988    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="62091dd9-fd32-4fa1-afd5-7ba971f2a163" containerName="revoke-keys"
Nov 04 00:15:00.144351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.144327    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"]
Nov 04 00:15:00.144521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.144503    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.150139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.150108    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"]
Nov 04 00:15:00.151871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.151846    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.154890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.154864    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:15:00.158920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.158896    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"]
Nov 04 00:15:00.159893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.159139    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.182979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.182943    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"]
Nov 04 00:15:00.183822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.183796    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.207901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.207860    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszzs\" (UniqueName: \"kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.208212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.208187    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.208400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.208381    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.208631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.208582    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.208844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.208823    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.209046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.209024    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntzk\" (UniqueName: \"kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.210615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.210588    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5gk\" (UniqueName: \"kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.211004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.210981    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg89c\" (UniqueName: \"kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.211330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.209798    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.224766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.224704    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"]
Nov 04 00:15:00.241613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.241572    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"]
Nov 04 00:15:00.251518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.251490    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"]
Nov 04 00:15:00.256416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.255475    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"]
Nov 04 00:15:00.272808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.270031    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"]
Nov 04 00:15:00.272808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.270097    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.286075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.281632    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"]
Nov 04 00:15:00.286557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.286536    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.291973 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.291950    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"]
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.297845    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.306356    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"]
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.307437    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.312225    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313198    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313318    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313420    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313477    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntzk\" (UniqueName: \"kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313541    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313608    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5gk\" (UniqueName: \"kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313649    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313781    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzbl\" (UniqueName: \"kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313852    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg89c\" (UniqueName: \"kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313905    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313947    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsj6\" (UniqueName: \"kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.313979    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mps8\" (UniqueName: \"kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.314025    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.314058    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.314061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.314062    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2f58\" (UniqueName: \"kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.315380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.314312    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszzs\" (UniqueName: \"kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.315380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.315125    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.315380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.315181    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.315802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.315775    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.321277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.321256    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"]
Nov 04 00:15:00.321441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.321294    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"]
Nov 04 00:15:00.321441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.321365    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.321698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.321681    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"]
Nov 04 00:15:00.327413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.327391    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"]
Nov 04 00:15:00.334182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.334108    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"]
Nov 04 00:15:00.334182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.334125    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5gk\" (UniqueName: \"kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.334987 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.334959    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg89c\" (UniqueName: \"kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.335825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.335794    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszzs\" (UniqueName: \"kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.336882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.336857    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntzk\" (UniqueName: \"kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.338123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.338103    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"]
Nov 04 00:15:00.342548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.342529    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"]
Nov 04 00:15:00.347470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.347451    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"]
Nov 04 00:15:00.347607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.347589    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.352503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.352485    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"]
Nov 04 00:15:00.352624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.352604    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.361400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.361369    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.373940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.373820    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"]
Nov 04 00:15:00.385658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.381789    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"]
Nov 04 00:15:00.390823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.390712    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"]
Nov 04 00:15:00.401308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.401278    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.402114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.402079    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"]
Nov 04 00:15:00.408257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.408233    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"]
Nov 04 00:15:00.413707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.413686    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.417095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417054    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.417228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417195    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44js7\" (UniqueName: \"kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.417451 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417310    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895855-r8h7s\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.417451 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417439    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.417647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417479    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.417781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417720    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.417861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417828    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtpr\" (UniqueName: \"kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr\") pod \"maas-api-key-cleanup-29895855-r8h7s\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.417921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417878    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.417982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417950    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6qk\" (UniqueName: \"kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.418051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.417989    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzbl\" (UniqueName: \"kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.418112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418048    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfx7\" (UniqueName: \"kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.418171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418105    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tk46\" (UniqueName: \"kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.418171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418132    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvsk\" (UniqueName: \"kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.418171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418164    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.418342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418204    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsj6\" (UniqueName: \"kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.418342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418234    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mps8\" (UniqueName: \"kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.418342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418272    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.418342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418305    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f58\" (UniqueName: \"kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.418342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418337    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.418589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418367    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.418589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.418450    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.419248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.419218    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.419355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.419248    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"]
Nov 04 00:15:00.419355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.419255    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.419978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.419940    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.424417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.424393    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"]
Nov 04 00:15:00.428882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.428866    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"]
Nov 04 00:15:00.433618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.433591    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzbl\" (UniqueName: \"kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.433729 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.433704    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mps8\" (UniqueName: \"kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.433839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.433755    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2f58\" (UniqueName: \"kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.434313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.434292    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsj6\" (UniqueName: \"kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.434560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.434544    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"]
Nov 04 00:15:00.440033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.440015    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"]
Nov 04 00:15:00.440170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.440152    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.462018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.461979    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"]
Nov 04 00:15:00.462157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.462050    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"]
Nov 04 00:15:00.462157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.462153    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.468853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.468818    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"]
Nov 04 00:15:00.474959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.474939    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.484086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.484064    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"]
Nov 04 00:15:00.484252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.484232    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:00.513259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.513231    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:00.520086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520057    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.520240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520102    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.520240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520160    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.520240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520193    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtpr\" (UniqueName: \"kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr\") pod \"maas-api-key-cleanup-29895855-r8h7s\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.520240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520230    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2zj\" (UniqueName: \"kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.520614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520292    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k72sp\" (UniqueName: \"kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.520614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520324    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6qk\" (UniqueName: \"kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.520614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520384    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfx7\" (UniqueName: \"kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.520614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520438    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tk46\" (UniqueName: \"kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.520614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520463    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvsk\" (UniqueName: \"kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.520614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520508    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.521026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.520983    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.521498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521471    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.521679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521516    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.521679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521589    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.521679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521623    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.521679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521658    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5hg\" (UniqueName: \"kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.521679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521706    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44js7\" (UniqueName: \"kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.522430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521754    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.522430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521777    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.522430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521794    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895855-r8h7s\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.522430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.521862    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrbc\" (UniqueName: \"kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.523074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.522763    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.523074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.522799    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895855-r8h7s\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.523497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.523449    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.523497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.523458    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.534202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.534176    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tk46\" (UniqueName: \"kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.536193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.536168    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44js7\" (UniqueName: \"kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.537835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.537812    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtpr\" (UniqueName: \"kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr\") pod \"maas-api-key-cleanup-29895855-r8h7s\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.538793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.538765    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6qk\" (UniqueName: \"kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.540193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.540138    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvsk\" (UniqueName: \"kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.543146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.543092    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfx7\" (UniqueName: \"kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.558795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.557917    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:00.587193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.587162    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:00.592796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.592358    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:00.604097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.603903    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:00.623784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.623669    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrbc\" (UniqueName: \"kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.624813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.624140    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:00.627622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.625307    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.627622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.625413    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2zj\" (UniqueName: \"kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.627622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.627569    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.625473    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k72sp\" (UniqueName: \"kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.628032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.628133    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.628169    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5hg\" (UniqueName: \"kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.628220    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.628976    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.629609    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.630439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.629842    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.641539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.637257    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:00.641539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.638079    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrbc\" (UniqueName: \"kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.641539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.639125    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2zj\" (UniqueName: \"kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.646903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.646851    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5hg\" (UniqueName: \"kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.649290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.649245    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k72sp\" (UniqueName: \"kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:00.704380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.704154    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:00.716135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.715209    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:00.730419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.728886    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:00.737895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.736001    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:00.772130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.771879    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:00.782787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.782542    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:00.797775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.797524    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:00.809129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.809096    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"]
Nov 04 00:15:00.839972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.839865    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:00.850615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.850585    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:00.879413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:00.872210    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:01.064782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.062427    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"]
Nov 04 00:15:01.389202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.389098    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"]
Nov 04 00:15:01.395734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.395633    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"]
Nov 04 00:15:01.400047 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:01.400005    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43a2226_93f4_46a6_94ca_db9ec3552ac0.slice/crio-8848894a208ea3c5c17ed43021d013a20d0959ffe422360c8e9de54eeffbe09b WatchSource:0}: Error finding container 8848894a208ea3c5c17ed43021d013a20d0959ffe422360c8e9de54eeffbe09b: Status 404 returned error can't find the container with id 8848894a208ea3c5c17ed43021d013a20d0959ffe422360c8e9de54eeffbe09b
Nov 04 00:15:01.407884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.407763    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"]
Nov 04 00:15:01.415273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.415250    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"]
Nov 04 00:15:01.423788 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:01.423734    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc330931a_3ea9_4fe6_85fd_0a1d13c4a20f.slice/crio-6b3d6e39803a1a229c872c5c2ac431de839eed281396086241819effc2947087 WatchSource:0}: Error finding container 6b3d6e39803a1a229c872c5c2ac431de839eed281396086241819effc2947087: Status 404 returned error can't find the container with id 6b3d6e39803a1a229c872c5c2ac431de839eed281396086241819effc2947087
Nov 04 00:15:01.470552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.470515    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"]
Nov 04 00:15:01.507143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.507113    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"]
Nov 04 00:15:01.514552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.514523    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"]
Nov 04 00:15:01.588966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.588930    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"]
Nov 04 00:15:01.593798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.593772    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"]
Nov 04 00:15:01.608297 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:01.606292    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f28e54_550c_4e0f_b8a3_3a0fef812ae5.slice/crio-5515cf57d783295bfbf59a0bfc2cad5d2dc2a29c6eecf6699180fd5665fb4416 WatchSource:0}: Error finding container 5515cf57d783295bfbf59a0bfc2cad5d2dc2a29c6eecf6699180fd5665fb4416: Status 404 returned error can't find the container with id 5515cf57d783295bfbf59a0bfc2cad5d2dc2a29c6eecf6699180fd5665fb4416
Nov 04 00:15:01.612121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.612097    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"]
Nov 04 00:15:01.619504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.619479    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"]
Nov 04 00:15:01.625433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.625415    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"]
Nov 04 00:15:01.629577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.629556    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"]
Nov 04 00:15:01.656476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.656449    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"]
Nov 04 00:15:01.662578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.662550    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"]
Nov 04 00:15:01.712543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.712408    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"]
Nov 04 00:15:01.717575 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:01.717549    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff3587a_a841_4d1a_9151_e7045abe8c7b.slice/crio-cce830475e005fd933118990acd7d6b432a18dfe5c93cafc5a9d30e5e294ef67 WatchSource:0}: Error finding container cce830475e005fd933118990acd7d6b432a18dfe5c93cafc5a9d30e5e294ef67: Status 404 returned error can't find the container with id cce830475e005fd933118990acd7d6b432a18dfe5c93cafc5a9d30e5e294ef67
Nov 04 00:15:01.841493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.841457    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerStarted","Data":"5515cf57d783295bfbf59a0bfc2cad5d2dc2a29c6eecf6699180fd5665fb4416"}
Nov 04 00:15:01.842990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.842957    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" event={"ID":"f10b6447-61da-42f7-b173-474779526489","Type":"ContainerStarted","Data":"039cbd5d60d5fdfbca6d3310c33fdf66992e2d93bf334112570e1ebcc166663a"}
Nov 04 00:15:01.844414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.844386    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" event={"ID":"dedc8c18-693b-4c80-a315-abdf48d07bc3","Type":"ContainerStarted","Data":"fdf90b0285be53904b825981a884f9adadccabd408034d806a1ccc1325be2c0b"}
Nov 04 00:15:01.845685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.845660    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerStarted","Data":"6952fe2a7e364c1901ac5242311e7ca80defd389c9dfe7b1fed3307e6f76e786"}
Nov 04 00:15:01.846917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.846892    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerStarted","Data":"6b3d6e39803a1a229c872c5c2ac431de839eed281396086241819effc2947087"}
Nov 04 00:15:01.848247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.848224    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" event={"ID":"c43a2226-93f4-46a6-94ca-db9ec3552ac0","Type":"ContainerStarted","Data":"8848894a208ea3c5c17ed43021d013a20d0959ffe422360c8e9de54eeffbe09b"}
Nov 04 00:15:01.849652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.849625    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" event={"ID":"216a9882-650f-4a5d-96b5-414b34d235ac","Type":"ContainerStarted","Data":"54d0b155503b832b28627dbc495f527e48a5d50acf1f01ff2da7460b72a50611"}
Nov 04 00:15:01.851114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.851092    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" event={"ID":"862b2732-ead5-4f3a-9b68-881210dcb7e9","Type":"ContainerStarted","Data":"dfc3b5631742a5254b23240a30f9bf0176e49680e5e5dc50eb0fb69da630bc07"}
Nov 04 00:15:01.852414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.852386    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" event={"ID":"222b3305-bcdd-4f33-a787-7ff020d19e19","Type":"ContainerStarted","Data":"d939a75cf74e7e1ba9eff6600568c337a1c0c31090e2dd385f277ff651525134"}
Nov 04 00:15:01.854205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.854179    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" event={"ID":"870046cf-6a49-4993-a0d4-6941cfea5636","Type":"ContainerStarted","Data":"dac5f5c9af4494ec2c6750f584736918fc306abca1c7f98e123d56a6212e5b72"}
Nov 04 00:15:01.854317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.854225    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" event={"ID":"870046cf-6a49-4993-a0d4-6941cfea5636","Type":"ContainerStarted","Data":"0ca75e3bb81afcd6f0d26eaecdd533914cf7188d1e58db8ebab29e0f27e0c166"}
Nov 04 00:15:01.855597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.855571    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" event={"ID":"5030efe2-f18e-4640-88f2-c65df1984cfc","Type":"ContainerStarted","Data":"4ee2b1febb107f3e60666154da2b2531e47025ce52b45c8486a6aa307873a62b"}
Nov 04 00:15:01.857863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.857613    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" event={"ID":"4d1fb445-cd3b-465f-bf6e-5a098acb44b7","Type":"ContainerStarted","Data":"f733b5a3cbd190cb3f34ea553de76dd8cc526fa459d33a1af736cee6af117c85"}
Nov 04 00:15:01.858076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.858048    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" event={"ID":"4d1fb445-cd3b-465f-bf6e-5a098acb44b7","Type":"ContainerStarted","Data":"da49dfdeaef41472a014a151f50ceb2278e26a0d0f4b3d56718980a861460984"}
Nov 04 00:15:01.859595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.859570    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" event={"ID":"b3190980-0a5b-40e0-9961-b3f16873b79f","Type":"ContainerStarted","Data":"d42fcb61a911f667cb56b8f7a37650ef33f34351c4f15d34c697098ef449bb19"}
Nov 04 00:15:01.861225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.861201    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" event={"ID":"4d713377-aee5-467f-a7c2-c9131a33f61e","Type":"ContainerStarted","Data":"51a64e06ccb72003ac6a7c26f8611a11f125898bb53f75ec5c175b6bb31bdb3a"}
Nov 04 00:15:01.862469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.862449    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" event={"ID":"656ec7be-d09e-4148-9159-21e2be885498","Type":"ContainerStarted","Data":"720be0825fc9d56a078d838d0a8066ca1bfa20e3fabdab4ba059873fda4a2885"}
Nov 04 00:15:01.863779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.863731    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" event={"ID":"ac2af7b7-02d9-419d-803a-051f2e66ed0f","Type":"ContainerStarted","Data":"a965590ffd9b3a48b8c1cbf7f7f11f113cc721e6c88e9d67c045b9b2bcf2f9ef"}
Nov 04 00:15:01.865260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.865225    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerStarted","Data":"cce830475e005fd933118990acd7d6b432a18dfe5c93cafc5a9d30e5e294ef67"}
Nov 04 00:15:01.866609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.866583    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" event={"ID":"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb","Type":"ContainerStarted","Data":"73e336dd6646e0fe6dda515a08f1ef9acb211594843781542f1a492bd6287e3f"}
Nov 04 00:15:01.878910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.878865    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" podStartSLOduration=1.878853407 podStartE2EDuration="1.878853407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:01.874349798 +0000 UTC m=+2766.973949945" watchObservedRunningTime="2026-11-04 00:15:01.878853407 +0000 UTC m=+2766.978453551"
Nov 04 00:15:01.895136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:01.895089    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" podStartSLOduration=1.89507588 podStartE2EDuration="1.89507588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:01.892555431 +0000 UTC m=+2766.992155589" watchObservedRunningTime="2026-11-04 00:15:01.89507588 +0000 UTC m=+2766.994676017"
Nov 04 00:15:02.881050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.881007    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerStarted","Data":"571cbf85acf745c49553f0b9d8e85a5066f5078ba78c49709007401ce983e3e9"}
Nov 04 00:15:02.882892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.882860    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerStarted","Data":"ca5ae8972a1217f323162a98b0bf6769c82d80d40e75cab69ba608558c7a1bb6"}
Nov 04 00:15:02.884908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.884880    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" event={"ID":"c43a2226-93f4-46a6-94ca-db9ec3552ac0","Type":"ContainerStarted","Data":"d84066da4ff54e5e41fd78ae7dc5c9916dee4d338869e8c00525e4aea9d810a8"}
Nov 04 00:15:02.887666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.887637    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" event={"ID":"216a9882-650f-4a5d-96b5-414b34d235ac","Type":"ContainerStarted","Data":"0c4c46095d0b1a758ae04e1670cd59dbab08ae8c47e592efc3f9c0a3391d7396"}
Nov 04 00:15:02.889526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.889477    5074 generic.go:356] "Generic (PLEG): container finished" podID="870046cf-6a49-4993-a0d4-6941cfea5636" containerID="dac5f5c9af4494ec2c6750f584736918fc306abca1c7f98e123d56a6212e5b72" exitCode=60
Nov 04 00:15:02.889611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.889557    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" event={"ID":"870046cf-6a49-4993-a0d4-6941cfea5636","Type":"ContainerDied","Data":"dac5f5c9af4494ec2c6750f584736918fc306abca1c7f98e123d56a6212e5b72"}
Nov 04 00:15:02.889797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.889781    5074 scope.go:122] "RemoveContainer" containerID="dac5f5c9af4494ec2c6750f584736918fc306abca1c7f98e123d56a6212e5b72"
Nov 04 00:15:02.892276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.892255    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerID="f733b5a3cbd190cb3f34ea553de76dd8cc526fa459d33a1af736cee6af117c85" exitCode=60
Nov 04 00:15:02.892362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.892325    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" event={"ID":"4d1fb445-cd3b-465f-bf6e-5a098acb44b7","Type":"ContainerDied","Data":"f733b5a3cbd190cb3f34ea553de76dd8cc526fa459d33a1af736cee6af117c85"}
Nov 04 00:15:02.892584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.892570    5074 scope.go:122] "RemoveContainer" containerID="f733b5a3cbd190cb3f34ea553de76dd8cc526fa459d33a1af736cee6af117c85"
Nov 04 00:15:02.894578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.894454    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" event={"ID":"b3190980-0a5b-40e0-9961-b3f16873b79f","Type":"ContainerStarted","Data":"123a3c672667557d015a094b211eea883a84252eaab33f1bc650c25d69f1e863"}
Nov 04 00:15:02.896375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.896350    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" event={"ID":"4d713377-aee5-467f-a7c2-c9131a33f61e","Type":"ContainerStarted","Data":"0716d271397e93532688bf2ae4951e736663efa15b9ac9e3b830d4d08404efa3"}
Nov 04 00:15:02.898804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.898105    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerStarted","Data":"dba3e906f9519717d2570b1ca5ba6d06c75ad75a70bf4dc794f50682fdc12bd6"}
Nov 04 00:15:02.902257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.902233    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" event={"ID":"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb","Type":"ContainerStarted","Data":"46d2cef194acea2f773a4dc16408fb73b763379cb16ed012086384f5a6f47a3a"}
Nov 04 00:15:02.904979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.904917    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" podStartSLOduration=2.904902777 podStartE2EDuration="2.904902777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.899812821 +0000 UTC m=+2767.999412977" watchObservedRunningTime="2026-11-04 00:15:02.904902777 +0000 UTC m=+2768.004502922"
Nov 04 00:15:02.910983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.910961    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerStarted","Data":"fd60667f9c1836d4b58cffde95c6fe14fd863cc1449f781664628c6531e391f6"}
Nov 04 00:15:02.920473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.920429    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" podStartSLOduration=2.920420415 podStartE2EDuration="2.920420415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.9191291 +0000 UTC m=+2768.018729247" watchObservedRunningTime="2026-11-04 00:15:02.920420415 +0000 UTC m=+2768.020020581"
Nov 04 00:15:02.944036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.943966    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" podStartSLOduration=2.943947531 podStartE2EDuration="2.943947531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.938758332 +0000 UTC m=+2768.038358476" watchObservedRunningTime="2026-11-04 00:15:02.943947531 +0000 UTC m=+2768.043547676"
Nov 04 00:15:02.990125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:02.990057    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" podStartSLOduration=2.99003874 podStartE2EDuration="2.99003874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.989202976 +0000 UTC m=+2768.088803124" watchObservedRunningTime="2026-11-04 00:15:02.99003874 +0000 UTC m=+2768.089638885"
Nov 04 00:15:03.011229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.011171    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" podStartSLOduration=3.01115418 podStartE2EDuration="3.01115418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.008289972 +0000 UTC m=+2768.107890117" watchObservedRunningTime="2026-11-04 00:15:03.01115418 +0000 UTC m=+2768.110754326"
Nov 04 00:15:03.034954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.034904    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" podStartSLOduration=3.034889217 podStartE2EDuration="3.034889217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.031369043 +0000 UTC m=+2768.130969187" watchObservedRunningTime="2026-11-04 00:15:03.034889217 +0000 UTC m=+2768.134489405"
Nov 04 00:15:03.077280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.077178    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" podStartSLOduration=3.077163969 podStartE2EDuration="3.077163969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.072836611 +0000 UTC m=+2768.172436773" watchObservedRunningTime="2026-11-04 00:15:03.077163969 +0000 UTC m=+2768.176764142"
Nov 04 00:15:03.095307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.095252    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" podStartSLOduration=3.095239259 podStartE2EDuration="3.095239259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.091641153 +0000 UTC m=+2768.191241298" watchObservedRunningTime="2026-11-04 00:15:03.095239259 +0000 UTC m=+2768.194839410"
Nov 04 00:15:03.124680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.122029    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" podStartSLOduration=3.122009615 podStartE2EDuration="3.122009615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.112435029 +0000 UTC m=+2768.212035175" watchObservedRunningTime="2026-11-04 00:15:03.122009615 +0000 UTC m=+2768.221609738"
Nov 04 00:15:03.929733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.929700    5074 generic.go:356] "Generic (PLEG): container finished" podID="656ec7be-d09e-4148-9159-21e2be885498" containerID="621b041def09af17b2d21193fc5328889aa1d6a32451b1c7a3f0a5242ebf74db" exitCode=60
Nov 04 00:15:03.930223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.929803    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" event={"ID":"656ec7be-d09e-4148-9159-21e2be885498","Type":"ContainerDied","Data":"621b041def09af17b2d21193fc5328889aa1d6a32451b1c7a3f0a5242ebf74db"}
Nov 04 00:15:03.930223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.930205    5074 scope.go:122] "RemoveContainer" containerID="621b041def09af17b2d21193fc5328889aa1d6a32451b1c7a3f0a5242ebf74db"
Nov 04 00:15:03.931964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.931932    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" event={"ID":"ac2af7b7-02d9-419d-803a-051f2e66ed0f","Type":"ContainerDied","Data":"48512b81755c964bba57e52db5ca8f14296f030e0b7070139e5dbf9f47151808"}
Nov 04 00:15:03.932200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.931833    5074 generic.go:356] "Generic (PLEG): container finished" podID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerID="48512b81755c964bba57e52db5ca8f14296f030e0b7070139e5dbf9f47151808" exitCode=60
Nov 04 00:15:03.932297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.932265    5074 scope.go:122] "RemoveContainer" containerID="48512b81755c964bba57e52db5ca8f14296f030e0b7070139e5dbf9f47151808"
Nov 04 00:15:03.936399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.936376    5074 generic.go:356] "Generic (PLEG): container finished" podID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerID="dba3e906f9519717d2570b1ca5ba6d06c75ad75a70bf4dc794f50682fdc12bd6" exitCode=60
Nov 04 00:15:03.936460 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.936450    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerDied","Data":"dba3e906f9519717d2570b1ca5ba6d06c75ad75a70bf4dc794f50682fdc12bd6"}
Nov 04 00:15:03.936765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.936729    5074 scope.go:122] "RemoveContainer" containerID="dba3e906f9519717d2570b1ca5ba6d06c75ad75a70bf4dc794f50682fdc12bd6"
Nov 04 00:15:03.938489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.938469    5074 generic.go:356] "Generic (PLEG): container finished" podID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerID="46d2cef194acea2f773a4dc16408fb73b763379cb16ed012086384f5a6f47a3a" exitCode=60
Nov 04 00:15:03.938588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.938547    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" event={"ID":"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb","Type":"ContainerDied","Data":"46d2cef194acea2f773a4dc16408fb73b763379cb16ed012086384f5a6f47a3a"}
Nov 04 00:15:03.938833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.938814    5074 scope.go:122] "RemoveContainer" containerID="46d2cef194acea2f773a4dc16408fb73b763379cb16ed012086384f5a6f47a3a"
Nov 04 00:15:03.940573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.940551    5074 generic.go:356] "Generic (PLEG): container finished" podID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerID="fd60667f9c1836d4b58cffde95c6fe14fd863cc1449f781664628c6531e391f6" exitCode=60
Nov 04 00:15:03.940665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.940627    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerDied","Data":"fd60667f9c1836d4b58cffde95c6fe14fd863cc1449f781664628c6531e391f6"}
Nov 04 00:15:03.940873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.940856    5074 scope.go:122] "RemoveContainer" containerID="fd60667f9c1836d4b58cffde95c6fe14fd863cc1449f781664628c6531e391f6"
Nov 04 00:15:03.942706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.942678    5074 generic.go:356] "Generic (PLEG): container finished" podID="f10b6447-61da-42f7-b173-474779526489" containerID="2e6395d759ee44329f06990386665ef8b04f9dad6ff822d4bf429bc4fd1a9e4b" exitCode=60
Nov 04 00:15:03.942842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.942819    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" event={"ID":"f10b6447-61da-42f7-b173-474779526489","Type":"ContainerDied","Data":"2e6395d759ee44329f06990386665ef8b04f9dad6ff822d4bf429bc4fd1a9e4b"}
Nov 04 00:15:03.943282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.943207    5074 scope.go:122] "RemoveContainer" containerID="2e6395d759ee44329f06990386665ef8b04f9dad6ff822d4bf429bc4fd1a9e4b"
Nov 04 00:15:03.947397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.947372    5074 generic.go:356] "Generic (PLEG): container finished" podID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerID="d592e93f9ebcf63829f16126b244525ff2e47cb2a171980f4a91d47d50867392" exitCode=60
Nov 04 00:15:03.947517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.947470    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" event={"ID":"dedc8c18-693b-4c80-a315-abdf48d07bc3","Type":"ContainerDied","Data":"d592e93f9ebcf63829f16126b244525ff2e47cb2a171980f4a91d47d50867392"}
Nov 04 00:15:03.948132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.947800    5074 scope.go:122] "RemoveContainer" containerID="d592e93f9ebcf63829f16126b244525ff2e47cb2a171980f4a91d47d50867392"
Nov 04 00:15:03.955918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.955892    5074 generic.go:356] "Generic (PLEG): container finished" podID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerID="571cbf85acf745c49553f0b9d8e85a5066f5078ba78c49709007401ce983e3e9" exitCode=60
Nov 04 00:15:03.956068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.955981    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerDied","Data":"571cbf85acf745c49553f0b9d8e85a5066f5078ba78c49709007401ce983e3e9"}
Nov 04 00:15:03.956264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.956248    5074 scope.go:122] "RemoveContainer" containerID="571cbf85acf745c49553f0b9d8e85a5066f5078ba78c49709007401ce983e3e9"
Nov 04 00:15:03.958580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.958555    5074 generic.go:356] "Generic (PLEG): container finished" podID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerID="ca5ae8972a1217f323162a98b0bf6769c82d80d40e75cab69ba608558c7a1bb6" exitCode=60
Nov 04 00:15:03.958697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.958642    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerDied","Data":"ca5ae8972a1217f323162a98b0bf6769c82d80d40e75cab69ba608558c7a1bb6"}
Nov 04 00:15:03.959134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.958990    5074 scope.go:122] "RemoveContainer" containerID="ca5ae8972a1217f323162a98b0bf6769c82d80d40e75cab69ba608558c7a1bb6"
Nov 04 00:15:03.960951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.960917    5074 generic.go:356] "Generic (PLEG): container finished" podID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerID="d84066da4ff54e5e41fd78ae7dc5c9916dee4d338869e8c00525e4aea9d810a8" exitCode=60
Nov 04 00:15:03.961063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.961003    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" event={"ID":"c43a2226-93f4-46a6-94ca-db9ec3552ac0","Type":"ContainerDied","Data":"d84066da4ff54e5e41fd78ae7dc5c9916dee4d338869e8c00525e4aea9d810a8"}
Nov 04 00:15:03.961255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.961234    5074 scope.go:122] "RemoveContainer" containerID="d84066da4ff54e5e41fd78ae7dc5c9916dee4d338869e8c00525e4aea9d810a8"
Nov 04 00:15:03.963023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.963001    5074 generic.go:356] "Generic (PLEG): container finished" podID="216a9882-650f-4a5d-96b5-414b34d235ac" containerID="0c4c46095d0b1a758ae04e1670cd59dbab08ae8c47e592efc3f9c0a3391d7396" exitCode=60
Nov 04 00:15:03.963099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.963071    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" event={"ID":"216a9882-650f-4a5d-96b5-414b34d235ac","Type":"ContainerDied","Data":"0c4c46095d0b1a758ae04e1670cd59dbab08ae8c47e592efc3f9c0a3391d7396"}
Nov 04 00:15:03.963322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.963301    5074 scope.go:122] "RemoveContainer" containerID="0c4c46095d0b1a758ae04e1670cd59dbab08ae8c47e592efc3f9c0a3391d7396"
Nov 04 00:15:03.965832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.965809    5074 generic.go:356] "Generic (PLEG): container finished" podID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerID="7dc41f36d8727fd7e65c903ae0e72c0f0877a4bd7d22ebda362c6bacad4bd2d6" exitCode=60
Nov 04 00:15:03.966276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.966033    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" event={"ID":"862b2732-ead5-4f3a-9b68-881210dcb7e9","Type":"ContainerDied","Data":"7dc41f36d8727fd7e65c903ae0e72c0f0877a4bd7d22ebda362c6bacad4bd2d6"}
Nov 04 00:15:03.966526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.966502    5074 scope.go:122] "RemoveContainer" containerID="7dc41f36d8727fd7e65c903ae0e72c0f0877a4bd7d22ebda362c6bacad4bd2d6"
Nov 04 00:15:03.978945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.978920    5074 generic.go:356] "Generic (PLEG): container finished" podID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerID="b0d4fb804b60ec50fab4b31bc9cb4ac1512856877e595a243cc684664f1ca834" exitCode=60
Nov 04 00:15:03.979175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.979155    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" event={"ID":"222b3305-bcdd-4f33-a787-7ff020d19e19","Type":"ContainerDied","Data":"b0d4fb804b60ec50fab4b31bc9cb4ac1512856877e595a243cc684664f1ca834"}
Nov 04 00:15:03.979558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.979541    5074 scope.go:122] "RemoveContainer" containerID="b0d4fb804b60ec50fab4b31bc9cb4ac1512856877e595a243cc684664f1ca834"
Nov 04 00:15:03.981766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.981729    5074 generic.go:356] "Generic (PLEG): container finished" podID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerID="f1d4261d043eb00f708af97f9d4aba32b541f19381fe91ff0810eb5e007cba98" exitCode=60
Nov 04 00:15:03.981980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.981819    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" event={"ID":"5030efe2-f18e-4640-88f2-c65df1984cfc","Type":"ContainerDied","Data":"f1d4261d043eb00f708af97f9d4aba32b541f19381fe91ff0810eb5e007cba98"}
Nov 04 00:15:03.982096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.982076    5074 scope.go:122] "RemoveContainer" containerID="f1d4261d043eb00f708af97f9d4aba32b541f19381fe91ff0810eb5e007cba98"
Nov 04 00:15:03.988543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.988521    5074 generic.go:356] "Generic (PLEG): container finished" podID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerID="123a3c672667557d015a094b211eea883a84252eaab33f1bc650c25d69f1e863" exitCode=60
Nov 04 00:15:03.988661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.988553    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" event={"ID":"b3190980-0a5b-40e0-9961-b3f16873b79f","Type":"ContainerDied","Data":"123a3c672667557d015a094b211eea883a84252eaab33f1bc650c25d69f1e863"}
Nov 04 00:15:03.989362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.989341    5074 scope.go:122] "RemoveContainer" containerID="123a3c672667557d015a094b211eea883a84252eaab33f1bc650c25d69f1e863"
Nov 04 00:15:03.994009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.993979    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerID="0716d271397e93532688bf2ae4951e736663efa15b9ac9e3b830d4d08404efa3" exitCode=60
Nov 04 00:15:03.994098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.994019    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" event={"ID":"4d713377-aee5-467f-a7c2-c9131a33f61e","Type":"ContainerDied","Data":"0716d271397e93532688bf2ae4951e736663efa15b9ac9e3b830d4d08404efa3"}
Nov 04 00:15:03.994333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:03.994317    5074 scope.go:122] "RemoveContainer" containerID="0716d271397e93532688bf2ae4951e736663efa15b9ac9e3b830d4d08404efa3"
Nov 04 00:15:05.153204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.145341    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" event={"ID":"656ec7be-d09e-4148-9159-21e2be885498","Type":"ContainerStarted","Data":"ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a"}
Nov 04 00:15:05.167023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.166281    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" event={"ID":"ac2af7b7-02d9-419d-803a-051f2e66ed0f","Type":"ContainerStarted","Data":"c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2"}
Nov 04 00:15:05.170037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.168979    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerStarted","Data":"b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec"}
Nov 04 00:15:05.174774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.174091    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerStarted","Data":"533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b"}
Nov 04 00:15:05.177354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.176628    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" event={"ID":"f10b6447-61da-42f7-b173-474779526489","Type":"ContainerStarted","Data":"44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f"}
Nov 04 00:15:05.180974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.180570    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" event={"ID":"dedc8c18-693b-4c80-a315-abdf48d07bc3","Type":"ContainerStarted","Data":"95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916"}
Nov 04 00:15:05.187670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.187360    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerStarted","Data":"99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66"}
Nov 04 00:15:05.191980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.191145    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" event={"ID":"862b2732-ead5-4f3a-9b68-881210dcb7e9","Type":"ContainerStarted","Data":"08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc"}
Nov 04 00:15:05.191980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.191871    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" podStartSLOduration=5.191858594 podStartE2EDuration="5.191858594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:05.182241737 +0000 UTC m=+2770.281841882" watchObservedRunningTime="2026-11-04 00:15:05.191858594 +0000 UTC m=+2770.291458747"
Nov 04 00:15:05.194990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.194966    5074 generic.go:356] "Generic (PLEG): container finished" podID="870046cf-6a49-4993-a0d4-6941cfea5636" containerID="686c40f2707eb22d9d544f954ed62b6639c3cd67db660b009c110108288eaa87" exitCode=60
Nov 04 00:15:05.195107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.195092    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" event={"ID":"870046cf-6a49-4993-a0d4-6941cfea5636","Type":"ContainerDied","Data":"686c40f2707eb22d9d544f954ed62b6639c3cd67db660b009c110108288eaa87"}
Nov 04 00:15:05.195184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.195138    5074 scope.go:122] "RemoveContainer" containerID="dac5f5c9af4494ec2c6750f584736918fc306abca1c7f98e123d56a6212e5b72"
Nov 04 00:15:05.228660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.228630    5074 scope.go:122] "RemoveContainer" containerID="686c40f2707eb22d9d544f954ed62b6639c3cd67db660b009c110108288eaa87"
Nov 04 00:15:05.231355 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:05.231319    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2_odh-ai-gateway-infra(870046cf-6a49-4993-a0d4-6941cfea5636)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" podUID="870046cf-6a49-4993-a0d4-6941cfea5636"
Nov 04 00:15:05.233103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.233075    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-xm45v"]
Nov 04 00:15:05.233727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.233700    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" event={"ID":"5030efe2-f18e-4640-88f2-c65df1984cfc","Type":"ContainerStarted","Data":"4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f"}
Nov 04 00:15:05.245119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.245090    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerID="8d4804b55cc521b6b4714eb0badcf84bcd2ccedfcba46bc11d32f7e3349f95a1" exitCode=60
Nov 04 00:15:05.245278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.245222    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" event={"ID":"4d1fb445-cd3b-465f-bf6e-5a098acb44b7","Type":"ContainerDied","Data":"8d4804b55cc521b6b4714eb0badcf84bcd2ccedfcba46bc11d32f7e3349f95a1"}
Nov 04 00:15:05.245552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.245452    5074 scope.go:122] "RemoveContainer" containerID="8d4804b55cc521b6b4714eb0badcf84bcd2ccedfcba46bc11d32f7e3349f95a1"
Nov 04 00:15:05.249289 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:05.245786    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j_odh-ai-gateway-infra(4d1fb445-cd3b-465f-bf6e-5a098acb44b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7"
Nov 04 00:15:05.276123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.276100    5074 scope.go:122] "RemoveContainer" containerID="f733b5a3cbd190cb3f34ea553de76dd8cc526fa459d33a1af736cee6af117c85"
Nov 04 00:15:05.284206 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.284162    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" podStartSLOduration=5.284146814 podStartE2EDuration="5.284146814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:05.270732133 +0000 UTC m=+2770.370332278" watchObservedRunningTime="2026-11-04 00:15:05.284146814 +0000 UTC m=+2770.383746961"
Nov 04 00:15:05.293301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.293246    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" podStartSLOduration=5.293229827 podStartE2EDuration="5.293229827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:05.292154988 +0000 UTC m=+2770.391755126" watchObservedRunningTime="2026-11-04 00:15:05.293229827 +0000 UTC m=+2770.392829971"
Nov 04 00:15:05.322466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.322405    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" podStartSLOduration=5.322385867 podStartE2EDuration="5.322385867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:05.313149968 +0000 UTC m=+2770.412750112" watchObservedRunningTime="2026-11-04 00:15:05.322385867 +0000 UTC m=+2770.421986011"
Nov 04 00:15:05.372687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.372620    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" podStartSLOduration=5.372602822 podStartE2EDuration="5.372602822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:05.371688278 +0000 UTC m=+2770.471288421" watchObservedRunningTime="2026-11-04 00:15:05.372602822 +0000 UTC m=+2770.472202964"
Nov 04 00:15:05.599533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:05.598388    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" podStartSLOduration=5.598362945 podStartE2EDuration="5.598362945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:05.438533029 +0000 UTC m=+2770.538133174" watchObservedRunningTime="2026-11-04 00:15:05.598362945 +0000 UTC m=+2770.697963091"
Nov 04 00:15:06.257111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.257080    5074 generic.go:356] "Generic (PLEG): container finished" podID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerID="3aff2efb0fde425f87454eda4a38b7316fb2597ae489d16e219acc3fea4228f1" exitCode=60
Nov 04 00:15:06.257540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.257167    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" event={"ID":"c43a2226-93f4-46a6-94ca-db9ec3552ac0","Type":"ContainerDied","Data":"3aff2efb0fde425f87454eda4a38b7316fb2597ae489d16e219acc3fea4228f1"}
Nov 04 00:15:06.257540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.257244    5074 scope.go:122] "RemoveContainer" containerID="d84066da4ff54e5e41fd78ae7dc5c9916dee4d338869e8c00525e4aea9d810a8"
Nov 04 00:15:06.257540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.257470    5074 scope.go:122] "RemoveContainer" containerID="3aff2efb0fde425f87454eda4a38b7316fb2597ae489d16e219acc3fea4228f1"
Nov 04 00:15:06.257909 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.257881    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6_odh-ai-gateway-infra(c43a2226-93f4-46a6-94ca-db9ec3552ac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0"
Nov 04 00:15:06.259482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.259453    5074 generic.go:356] "Generic (PLEG): container finished" podID="216a9882-650f-4a5d-96b5-414b34d235ac" containerID="cfe217984155f1147f93c624d87075a934b845d16275369955c0043613d51df3" exitCode=60
Nov 04 00:15:06.259585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.259537    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" event={"ID":"216a9882-650f-4a5d-96b5-414b34d235ac","Type":"ContainerDied","Data":"cfe217984155f1147f93c624d87075a934b845d16275369955c0043613d51df3"}
Nov 04 00:15:06.259906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.259890    5074 scope.go:122] "RemoveContainer" containerID="cfe217984155f1147f93c624d87075a934b845d16275369955c0043613d51df3"
Nov 04 00:15:06.260272 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.260248    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf_odh-ai-gateway-infra(216a9882-650f-4a5d-96b5-414b34d235ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" podUID="216a9882-650f-4a5d-96b5-414b34d235ac"
Nov 04 00:15:06.261683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.261662    5074 generic.go:356] "Generic (PLEG): container finished" podID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerID="08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc" exitCode=60
Nov 04 00:15:06.261800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.261754    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" event={"ID":"862b2732-ead5-4f3a-9b68-881210dcb7e9","Type":"ContainerDied","Data":"08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc"}
Nov 04 00:15:06.262057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.262036    5074 scope.go:122] "RemoveContainer" containerID="08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc"
Nov 04 00:15:06.262448 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.262417    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797_odh-ai-gateway-infra(862b2732-ead5-4f3a-9b68-881210dcb7e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9"
Nov 04 00:15:06.263905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.263882    5074 generic.go:356] "Generic (PLEG): container finished" podID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerID="78de06fcb09d7f24615a15fdde675b7ba254bc1c648ca5ddf0bfd46ce17a5cfd" exitCode=60
Nov 04 00:15:06.263999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.263932    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" event={"ID":"222b3305-bcdd-4f33-a787-7ff020d19e19","Type":"ContainerDied","Data":"78de06fcb09d7f24615a15fdde675b7ba254bc1c648ca5ddf0bfd46ce17a5cfd"}
Nov 04 00:15:06.264233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.264214    5074 scope.go:122] "RemoveContainer" containerID="78de06fcb09d7f24615a15fdde675b7ba254bc1c648ca5ddf0bfd46ce17a5cfd"
Nov 04 00:15:06.264565 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.264545    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx_odh-ai-gateway-infra(222b3305-bcdd-4f33-a787-7ff020d19e19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19"
Nov 04 00:15:06.266137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.266111    5074 scope.go:122] "RemoveContainer" containerID="686c40f2707eb22d9d544f954ed62b6639c3cd67db660b009c110108288eaa87"
Nov 04 00:15:06.266403 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.266384    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2_odh-ai-gateway-infra(870046cf-6a49-4993-a0d4-6941cfea5636)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" podUID="870046cf-6a49-4993-a0d4-6941cfea5636"
Nov 04 00:15:06.267705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.267685    5074 generic.go:356] "Generic (PLEG): container finished" podID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerID="4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f" exitCode=60
Nov 04 00:15:06.267824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.267764    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" event={"ID":"5030efe2-f18e-4640-88f2-c65df1984cfc","Type":"ContainerDied","Data":"4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f"}
Nov 04 00:15:06.268044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.268027    5074 scope.go:122] "RemoveContainer" containerID="4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f"
Nov 04 00:15:06.268359 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.268338    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm_odh-ai-gateway-infra(5030efe2-f18e-4640-88f2-c65df1984cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc"
Nov 04 00:15:06.269602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.269581    5074 scope.go:122] "RemoveContainer" containerID="8d4804b55cc521b6b4714eb0badcf84bcd2ccedfcba46bc11d32f7e3349f95a1"
Nov 04 00:15:06.269896 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.269870    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j_odh-ai-gateway-infra(4d1fb445-cd3b-465f-bf6e-5a098acb44b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7"
Nov 04 00:15:06.271459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.271436    5074 generic.go:356] "Generic (PLEG): container finished" podID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerID="9dbd1e160217b12aba87fed94720be0ee12933fa81934d55c234e62e9e832757" exitCode=60
Nov 04 00:15:06.271548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.271515    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" event={"ID":"b3190980-0a5b-40e0-9961-b3f16873b79f","Type":"ContainerDied","Data":"9dbd1e160217b12aba87fed94720be0ee12933fa81934d55c234e62e9e832757"}
Nov 04 00:15:06.271712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.271698    5074 scope.go:122] "RemoveContainer" containerID="9dbd1e160217b12aba87fed94720be0ee12933fa81934d55c234e62e9e832757"
Nov 04 00:15:06.272002 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.271979    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298_odh-ai-gateway-infra(b3190980-0a5b-40e0-9961-b3f16873b79f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f"
Nov 04 00:15:06.273434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.273413    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerID="966c0396e1b1b16e05fac56771aeafffc1d374960783575f984e652dc3d23955" exitCode=60
Nov 04 00:15:06.273534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.273496    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" event={"ID":"4d713377-aee5-467f-a7c2-c9131a33f61e","Type":"ContainerDied","Data":"966c0396e1b1b16e05fac56771aeafffc1d374960783575f984e652dc3d23955"}
Nov 04 00:15:06.273752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.273723    5074 scope.go:122] "RemoveContainer" containerID="966c0396e1b1b16e05fac56771aeafffc1d374960783575f984e652dc3d23955"
Nov 04 00:15:06.274041 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.274017    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4_odh-ai-gateway-infra(4d713377-aee5-467f-a7c2-c9131a33f61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e"
Nov 04 00:15:06.275476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.275459    5074 generic.go:356] "Generic (PLEG): container finished" podID="656ec7be-d09e-4148-9159-21e2be885498" containerID="ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a" exitCode=60
Nov 04 00:15:06.275571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.275539    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" event={"ID":"656ec7be-d09e-4148-9159-21e2be885498","Type":"ContainerDied","Data":"ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a"}
Nov 04 00:15:06.275771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.275722    5074 scope.go:122] "RemoveContainer" containerID="ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a"
Nov 04 00:15:06.276001 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.275978    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd_odh-ai-gateway-infra(656ec7be-d09e-4148-9159-21e2be885498)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" podUID="656ec7be-d09e-4148-9159-21e2be885498"
Nov 04 00:15:06.277474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.277403    5074 generic.go:356] "Generic (PLEG): container finished" podID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerID="c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2" exitCode=60
Nov 04 00:15:06.277474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.277462    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" event={"ID":"ac2af7b7-02d9-419d-803a-051f2e66ed0f","Type":"ContainerDied","Data":"c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2"}
Nov 04 00:15:06.277804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.277784    5074 scope.go:122] "RemoveContainer" containerID="c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2"
Nov 04 00:15:06.278594 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.278563    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf_odh-ai-gateway-infra(ac2af7b7-02d9-419d-803a-051f2e66ed0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f"
Nov 04 00:15:06.280302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.280273    5074 generic.go:356] "Generic (PLEG): container finished" podID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerID="b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec" exitCode=60
Nov 04 00:15:06.280391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.280344    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerDied","Data":"b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec"}
Nov 04 00:15:06.280622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.280603    5074 scope.go:122] "RemoveContainer" containerID="b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec"
Nov 04 00:15:06.280894 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.280869    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895855-r8h7s_odh-ai-gateway-infra(aff3587a-a841-4d1a-9151-e7045abe8c7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b"
Nov 04 00:15:06.284135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.283488    5074 generic.go:356] "Generic (PLEG): container finished" podID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerID="a3b9b7fa79ffb08fdf1f1fcedd554b197ca4a37380e7489b867da5ad5aead702" exitCode=60
Nov 04 00:15:06.284135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.283656    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" event={"ID":"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb","Type":"ContainerDied","Data":"a3b9b7fa79ffb08fdf1f1fcedd554b197ca4a37380e7489b867da5ad5aead702"}
Nov 04 00:15:06.284135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.283851    5074 scope.go:122] "RemoveContainer" containerID="a3b9b7fa79ffb08fdf1f1fcedd554b197ca4a37380e7489b867da5ad5aead702"
Nov 04 00:15:06.284135 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.284092    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p_odh-ai-gateway-infra(6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb"
Nov 04 00:15:06.286986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.286969    5074 scope.go:122] "RemoveContainer" containerID="0c4c46095d0b1a758ae04e1670cd59dbab08ae8c47e592efc3f9c0a3391d7396"
Nov 04 00:15:06.287053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.287022    5074 generic.go:356] "Generic (PLEG): container finished" podID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerID="533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b" exitCode=60
Nov 04 00:15:06.287110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.287090    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerDied","Data":"533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b"}
Nov 04 00:15:06.287309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.287295    5074 scope.go:122] "RemoveContainer" containerID="533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b"
Nov 04 00:15:06.287516 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.287491    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw_odh-ai-gateway-infra(a9f28e54-550c-4e0f-b8a3-3a0fef812ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5"
Nov 04 00:15:06.289290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.289271    5074 generic.go:356] "Generic (PLEG): container finished" podID="f10b6447-61da-42f7-b173-474779526489" containerID="44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f" exitCode=60
Nov 04 00:15:06.289398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.289356    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" event={"ID":"f10b6447-61da-42f7-b173-474779526489","Type":"ContainerDied","Data":"44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f"}
Nov 04 00:15:06.289549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.289533    5074 scope.go:122] "RemoveContainer" containerID="44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f"
Nov 04 00:15:06.289809 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.289784    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn_odh-ai-gateway-infra(f10b6447-61da-42f7-b173-474779526489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" podUID="f10b6447-61da-42f7-b173-474779526489"
Nov 04 00:15:06.291217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.291198    5074 generic.go:356] "Generic (PLEG): container finished" podID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerID="95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916" exitCode=60
Nov 04 00:15:06.291316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.291277    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" event={"ID":"dedc8c18-693b-4c80-a315-abdf48d07bc3","Type":"ContainerDied","Data":"95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916"}
Nov 04 00:15:06.291506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.291487    5074 scope.go:122] "RemoveContainer" containerID="95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916"
Nov 04 00:15:06.291824 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.291799    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l_odh-ai-gateway-infra(dedc8c18-693b-4c80-a315-abdf48d07bc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3"
Nov 04 00:15:06.293487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.293448    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerStarted","Data":"ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e"}
Nov 04 00:15:06.297357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.297286    5074 generic.go:356] "Generic (PLEG): container finished" podID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerID="99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66" exitCode=60
Nov 04 00:15:06.297357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.297324    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerDied","Data":"99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66"}
Nov 04 00:15:06.297820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.297799    5074 scope.go:122] "RemoveContainer" containerID="99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66"
Nov 04 00:15:06.298026 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:06.297997    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2_odh-ai-gateway-infra(c330931a-3ea9-4fe6-85fd-0a1d13c4a20f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f"
Nov 04 00:15:06.311673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.311651    5074 scope.go:122] "RemoveContainer" containerID="7dc41f36d8727fd7e65c903ae0e72c0f0877a4bd7d22ebda362c6bacad4bd2d6"
Nov 04 00:15:06.362218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.362190    5074 scope.go:122] "RemoveContainer" containerID="b0d4fb804b60ec50fab4b31bc9cb4ac1512856877e595a243cc684664f1ca834"
Nov 04 00:15:06.385638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.385586    5074 scope.go:122] "RemoveContainer" containerID="f1d4261d043eb00f708af97f9d4aba32b541f19381fe91ff0810eb5e007cba98"
Nov 04 00:15:06.403144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.403118    5074 scope.go:122] "RemoveContainer" containerID="123a3c672667557d015a094b211eea883a84252eaab33f1bc650c25d69f1e863"
Nov 04 00:15:06.418648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.418622    5074 scope.go:122] "RemoveContainer" containerID="0716d271397e93532688bf2ae4951e736663efa15b9ac9e3b830d4d08404efa3"
Nov 04 00:15:06.435566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.435543    5074 scope.go:122] "RemoveContainer" containerID="621b041def09af17b2d21193fc5328889aa1d6a32451b1c7a3f0a5242ebf74db"
Nov 04 00:15:06.450972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.450951    5074 scope.go:122] "RemoveContainer" containerID="48512b81755c964bba57e52db5ca8f14296f030e0b7070139e5dbf9f47151808"
Nov 04 00:15:06.466601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.466578    5074 scope.go:122] "RemoveContainer" containerID="dba3e906f9519717d2570b1ca5ba6d06c75ad75a70bf4dc794f50682fdc12bd6"
Nov 04 00:15:06.509605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.509573    5074 scope.go:122] "RemoveContainer" containerID="46d2cef194acea2f773a4dc16408fb73b763379cb16ed012086384f5a6f47a3a"
Nov 04 00:15:06.528791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.528767    5074 scope.go:122] "RemoveContainer" containerID="fd60667f9c1836d4b58cffde95c6fe14fd863cc1449f781664628c6531e391f6"
Nov 04 00:15:06.560703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.560680    5074 scope.go:122] "RemoveContainer" containerID="2e6395d759ee44329f06990386665ef8b04f9dad6ff822d4bf429bc4fd1a9e4b"
Nov 04 00:15:06.575853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.575833    5074 scope.go:122] "RemoveContainer" containerID="d592e93f9ebcf63829f16126b244525ff2e47cb2a171980f4a91d47d50867392"
Nov 04 00:15:06.591125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:06.591106    5074 scope.go:122] "RemoveContainer" containerID="ca5ae8972a1217f323162a98b0bf6769c82d80d40e75cab69ba608558c7a1bb6"
Nov 04 00:15:07.310559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.310528    5074 scope.go:122] "RemoveContainer" containerID="95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916"
Nov 04 00:15:07.311070 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.310936    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l_odh-ai-gateway-infra(dedc8c18-693b-4c80-a315-abdf48d07bc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3"
Nov 04 00:15:07.317985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.317885    5074 generic.go:356] "Generic (PLEG): container finished" podID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerID="ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e" exitCode=60
Nov 04 00:15:07.317985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.317955    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerDied","Data":"ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e"}
Nov 04 00:15:07.318205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.318025    5074 scope.go:122] "RemoveContainer" containerID="571cbf85acf745c49553f0b9d8e85a5066f5078ba78c49709007401ce983e3e9"
Nov 04 00:15:07.318324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.318304    5074 scope.go:122] "RemoveContainer" containerID="ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e"
Nov 04 00:15:07.318616 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.318593    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg_odh-ai-gateway-infra(093d3a77-e009-4e3d-aa72-3a7488d8e295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295"
Nov 04 00:15:07.320690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.320662    5074 scope.go:122] "RemoveContainer" containerID="99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66"
Nov 04 00:15:07.321072 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.321008    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2_odh-ai-gateway-infra(c330931a-3ea9-4fe6-85fd-0a1d13c4a20f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f"
Nov 04 00:15:07.322526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.322505    5074 scope.go:122] "RemoveContainer" containerID="3aff2efb0fde425f87454eda4a38b7316fb2597ae489d16e219acc3fea4228f1"
Nov 04 00:15:07.322836 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.322815    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6_odh-ai-gateway-infra(c43a2226-93f4-46a6-94ca-db9ec3552ac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0"
Nov 04 00:15:07.324217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.324197    5074 scope.go:122] "RemoveContainer" containerID="cfe217984155f1147f93c624d87075a934b845d16275369955c0043613d51df3"
Nov 04 00:15:07.324441 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.324419    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf_odh-ai-gateway-infra(216a9882-650f-4a5d-96b5-414b34d235ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" podUID="216a9882-650f-4a5d-96b5-414b34d235ac"
Nov 04 00:15:07.326312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.326293    5074 scope.go:122] "RemoveContainer" containerID="08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc"
Nov 04 00:15:07.326530 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.326507    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797_odh-ai-gateway-infra(862b2732-ead5-4f3a-9b68-881210dcb7e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9"
Nov 04 00:15:07.328186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.328166    5074 scope.go:122] "RemoveContainer" containerID="78de06fcb09d7f24615a15fdde675b7ba254bc1c648ca5ddf0bfd46ce17a5cfd"
Nov 04 00:15:07.328429 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.328402    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx_odh-ai-gateway-infra(222b3305-bcdd-4f33-a787-7ff020d19e19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19"
Nov 04 00:15:07.331182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.331164    5074 scope.go:122] "RemoveContainer" containerID="4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f"
Nov 04 00:15:07.331731 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.331401    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm_odh-ai-gateway-infra(5030efe2-f18e-4640-88f2-c65df1984cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc"
Nov 04 00:15:07.333636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.333619    5074 scope.go:122] "RemoveContainer" containerID="9dbd1e160217b12aba87fed94720be0ee12933fa81934d55c234e62e9e832757"
Nov 04 00:15:07.333920 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.333896    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298_odh-ai-gateway-infra(b3190980-0a5b-40e0-9961-b3f16873b79f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f"
Nov 04 00:15:07.338060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.335731    5074 scope.go:122] "RemoveContainer" containerID="966c0396e1b1b16e05fac56771aeafffc1d374960783575f984e652dc3d23955"
Nov 04 00:15:07.338060 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.336006    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4_odh-ai-gateway-infra(4d713377-aee5-467f-a7c2-c9131a33f61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e"
Nov 04 00:15:07.338060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.337624    5074 scope.go:122] "RemoveContainer" containerID="ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a"
Nov 04 00:15:07.338060 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.337820    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd_odh-ai-gateway-infra(656ec7be-d09e-4148-9159-21e2be885498)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" podUID="656ec7be-d09e-4148-9159-21e2be885498"
Nov 04 00:15:07.364337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.364306    5074 scope.go:122] "RemoveContainer" containerID="c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2"
Nov 04 00:15:07.364796 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.364728    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf_odh-ai-gateway-infra(ac2af7b7-02d9-419d-803a-051f2e66ed0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f"
Nov 04 00:15:07.366863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.366831    5074 scope.go:122] "RemoveContainer" containerID="b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec"
Nov 04 00:15:07.367523 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.367497    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895855-r8h7s_odh-ai-gateway-infra(aff3587a-a841-4d1a-9151-e7045abe8c7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b"
Nov 04 00:15:07.369963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.369933    5074 scope.go:122] "RemoveContainer" containerID="a3b9b7fa79ffb08fdf1f1fcedd554b197ca4a37380e7489b867da5ad5aead702"
Nov 04 00:15:07.370310 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.370283    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p_odh-ai-gateway-infra(6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb"
Nov 04 00:15:07.372251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.372221    5074 scope.go:122] "RemoveContainer" containerID="533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b"
Nov 04 00:15:07.372644 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.372617    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw_odh-ai-gateway-infra(a9f28e54-550c-4e0f-b8a3-3a0fef812ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5"
Nov 04 00:15:07.374060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:07.374042    5074 scope.go:122] "RemoveContainer" containerID="44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f"
Nov 04 00:15:07.374368 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:07.374349    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn_odh-ai-gateway-infra(f10b6447-61da-42f7-b173-474779526489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" podUID="f10b6447-61da-42f7-b173-474779526489"
Nov 04 00:15:08.387030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:08.386998    5074 scope.go:122] "RemoveContainer" containerID="ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e"
Nov 04 00:15:08.387450 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:08.387424    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg_odh-ai-gateway-infra(093d3a77-e009-4e3d-aa72-3a7488d8e295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295"
Nov 04 00:15:09.029662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:09.029623    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"]
Nov 04 00:15:09.036666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:09.036631    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac8d69a-f2grz"]
Nov 04 00:15:10.233583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.233532    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v" podUID="d09c3065-5cd0-407b-9adb-8fb78e8047cb" containerName="payload-processing" containerID="cri-o://c559b14d85d66339de8bd739767a35c07a1035d6c7a2e1c9a1c3d289afbf2f3c" gracePeriod=25
Nov 04 00:15:10.415317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.415044    5074 generic.go:356] "Generic (PLEG): container finished" podID="d09c3065-5cd0-407b-9adb-8fb78e8047cb" containerID="c559b14d85d66339de8bd739767a35c07a1035d6c7a2e1c9a1c3d289afbf2f3c" exitCode=0
Nov 04 00:15:10.415317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.415087    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v" event={"ID":"d09c3065-5cd0-407b-9adb-8fb78e8047cb","Type":"ContainerDied","Data":"c559b14d85d66339de8bd739767a35c07a1035d6c7a2e1c9a1c3d289afbf2f3c"}
Nov 04 00:15:10.546615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.546583    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:15:10.563860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.563831    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46\") pod \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") "
Nov 04 00:15:10.564035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.563939    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume\") pod \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") "
Nov 04 00:15:10.564035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.563986    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials\") pod \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") "
Nov 04 00:15:10.564035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.564017    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume\") pod \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\" (UID: \"d09c3065-5cd0-407b-9adb-8fb78e8047cb\") "
Nov 04 00:15:10.564392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.564349    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume" pod "d09c3065-5cd0-407b-9adb-8fb78e8047cb" (UID: "d09c3065-5cd0-407b-9adb-8fb78e8047cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:10.564484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.564427    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume" pod "d09c3065-5cd0-407b-9adb-8fb78e8047cb" (UID: "d09c3065-5cd0-407b-9adb-8fb78e8047cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:10.564725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.564697    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:10.564836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.564726    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c3065-5cd0-407b-9adb-8fb78e8047cb-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:10.567077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.567048    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials" pod "d09c3065-5cd0-407b-9adb-8fb78e8047cb" (UID: "d09c3065-5cd0-407b-9adb-8fb78e8047cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:10.567247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.567221    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46" pod "d09c3065-5cd0-407b-9adb-8fb78e8047cb" (UID: "d09c3065-5cd0-407b-9adb-8fb78e8047cb"). InnerVolumeSpecName "kube-api-access-z8d46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:10.666149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.666102    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:10.666149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:10.666145    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8d46\" (UniqueName: \"kubernetes.io/projected/d09c3065-5cd0-407b-9adb-8fb78e8047cb-kube-api-access-z8d46\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:11.025367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:11.025326    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c51d89-7648-49ae-87fd-f6970459e180" path="/var/lib/kubelet/pods/61c51d89-7648-49ae-87fd-f6970459e180/volumes"
Nov 04 00:15:11.429067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:11.428989    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v"
Nov 04 00:15:11.429552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:11.429023    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f897f846-xm45v" event={"ID":"d09c3065-5cd0-407b-9adb-8fb78e8047cb","Type":"ContainerDied","Data":"9c4f402ee53848c77c6f89f006785f1e281f3f4a3b9d4cfb2d364eb00b7ddd2a"}
Nov 04 00:15:11.429680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:11.429596    5074 scope.go:122] "RemoveContainer" containerID="c559b14d85d66339de8bd739767a35c07a1035d6c7a2e1c9a1c3d289afbf2f3c"
Nov 04 00:15:11.466000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:11.465966    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-xm45v"]
Nov 04 00:15:11.471014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:11.470984    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f897f846-xm45v"]
Nov 04 00:15:12.914926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.914881    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"]
Nov 04 00:15:12.916025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.915998    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d09c3065-5cd0-407b-9adb-8fb78e8047cb" containerName="payload-processing"
Nov 04 00:15:12.916025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.916022    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09c3065-5cd0-407b-9adb-8fb78e8047cb" containerName="payload-processing"
Nov 04 00:15:12.916242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.916190    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09c3065-5cd0-407b-9adb-8fb78e8047cb" containerName="payload-processing"
Nov 04 00:15:12.924302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.924274    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:12.930814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.930783    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"]
Nov 04 00:15:12.930975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.930890    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-11301e\""
Nov 04 00:15:12.997222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:12.997188    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"]
Nov 04 00:15:13.002699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.002657    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.002914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.002823    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tpl\" (UniqueName: \"kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl\") pod \"payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.005851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.005824    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.009861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.009837    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"]
Nov 04 00:15:13.012384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.012363    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-11301e-dockercfg-nq7h9\""
Nov 04 00:15:13.029133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.029100    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09c3065-5cd0-407b-9adb-8fb78e8047cb" path="/var/lib/kubelet/pods/d09c3065-5cd0-407b-9adb-8fb78e8047cb/volumes"
Nov 04 00:15:13.127760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.126761    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.127760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.126850    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tpl\" (UniqueName: \"kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl\") pod \"payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.127760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.126899    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.127760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.127006    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s652k\" (UniqueName: \"kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k\") pod \"payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.146771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.128227    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.193760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.184591    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tpl\" (UniqueName: \"kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl\") pod \"payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.228590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.228548    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.228825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.228631    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s652k\" (UniqueName: \"kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k\") pod \"payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.229374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.229347    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.242555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.242370    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s652k\" (UniqueName: \"kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k\") pod \"payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.244069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.244034    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:13.347392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.347107    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:13.552666 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:13.552631    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0c9be1_08ce_4e46_bc72_1605dce70b7d.slice/crio-855acbcdbc4c5f98d40a7514e7f0b89b9f6b4d0c9c70eff27bd451d0b3f08fe3 WatchSource:0}: Error finding container 855acbcdbc4c5f98d40a7514e7f0b89b9f6b4d0c9c70eff27bd451d0b3f08fe3: Status 404 returned error can't find the container with id 855acbcdbc4c5f98d40a7514e7f0b89b9f6b4d0c9c70eff27bd451d0b3f08fe3
Nov 04 00:15:13.553078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.553048    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"]
Nov 04 00:15:13.675087 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:13.675046    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a897f01_7e97_40d6_9e16_70deeede895f.slice/crio-bec270c16f9a0cb38809f025147636f0b49e5c0e7ee5bb5102f82818451fef1d WatchSource:0}: Error finding container bec270c16f9a0cb38809f025147636f0b49e5c0e7ee5bb5102f82818451fef1d: Status 404 returned error can't find the container with id bec270c16f9a0cb38809f025147636f0b49e5c0e7ee5bb5102f82818451fef1d
Nov 04 00:15:13.675986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:13.675962    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"]
Nov 04 00:15:14.466438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.466394    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" event={"ID":"0a897f01-7e97-40d6-9e16-70deeede895f","Type":"ContainerStarted","Data":"d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879"}
Nov 04 00:15:14.466984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.466460    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" event={"ID":"0a897f01-7e97-40d6-9e16-70deeede895f","Type":"ContainerStarted","Data":"bec270c16f9a0cb38809f025147636f0b49e5c0e7ee5bb5102f82818451fef1d"}
Nov 04 00:15:14.466984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.466494    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:14.468255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.468226    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" event={"ID":"7f0c9be1-08ce-4e46-bc72-1605dce70b7d","Type":"ContainerStarted","Data":"c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c"}
Nov 04 00:15:14.468383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.468280    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" event={"ID":"7f0c9be1-08ce-4e46-bc72-1605dce70b7d","Type":"ContainerStarted","Data":"855acbcdbc4c5f98d40a7514e7f0b89b9f6b4d0c9c70eff27bd451d0b3f08fe3"}
Nov 04 00:15:14.468383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.468312    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:14.488933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.488877    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" podStartSLOduration=2.4888602 podStartE2EDuration="2.4888602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:14.487486808 +0000 UTC m=+2779.587086954" watchObservedRunningTime="2026-11-04 00:15:14.4888602 +0000 UTC m=+2779.588460345"
Nov 04 00:15:14.517838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:14.517775    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" podStartSLOduration=2.517754046 podStartE2EDuration="2.517754046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:14.507619091 +0000 UTC m=+2779.607219235" watchObservedRunningTime="2026-11-04 00:15:14.517754046 +0000 UTC m=+2779.617354191"
Nov 04 00:15:17.010828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:17.010715    5074 scope.go:122] "RemoveContainer" containerID="966c0396e1b1b16e05fac56771aeafffc1d374960783575f984e652dc3d23955"
Nov 04 00:15:17.012707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:17.011957    5074 scope.go:122] "RemoveContainer" containerID="a3b9b7fa79ffb08fdf1f1fcedd554b197ca4a37380e7489b867da5ad5aead702"
Nov 04 00:15:17.015280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:17.014865    5074 scope.go:122] "RemoveContainer" containerID="4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f"
Nov 04 00:15:17.015447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:17.015417    5074 scope.go:122] "RemoveContainer" containerID="533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b"
Nov 04 00:15:18.010269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.010220    5074 scope.go:122] "RemoveContainer" containerID="95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916"
Nov 04 00:15:18.010509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.010481    5074 scope.go:122] "RemoveContainer" containerID="b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec"
Nov 04 00:15:18.010676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.010648    5074 scope.go:122] "RemoveContainer" containerID="c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2"
Nov 04 00:15:18.010851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.010829    5074 scope.go:122] "RemoveContainer" containerID="44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f"
Nov 04 00:15:18.011223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.011206    5074 scope.go:122] "RemoveContainer" containerID="ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a"
Nov 04 00:15:18.011524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.011505    5074 scope.go:122] "RemoveContainer" containerID="8d4804b55cc521b6b4714eb0badcf84bcd2ccedfcba46bc11d32f7e3349f95a1"
Nov 04 00:15:18.507804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.507725    5074 generic.go:356] "Generic (PLEG): container finished" podID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerID="c2d585471d79faa3e8a40d16e47102da4618966d03ee593063c4ce14f78b6344" exitCode=60
Nov 04 00:15:18.508003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.507791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" event={"ID":"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb","Type":"ContainerDied","Data":"c2d585471d79faa3e8a40d16e47102da4618966d03ee593063c4ce14f78b6344"}
Nov 04 00:15:18.509827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.507910    5074 scope.go:122] "RemoveContainer" containerID="a3b9b7fa79ffb08fdf1f1fcedd554b197ca4a37380e7489b867da5ad5aead702"
Nov 04 00:15:18.509962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.509949    5074 scope.go:122] "RemoveContainer" containerID="c2d585471d79faa3e8a40d16e47102da4618966d03ee593063c4ce14f78b6344"
Nov 04 00:15:18.510842 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:18.510437    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p_odh-ai-gateway-infra(6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb"
Nov 04 00:15:18.515717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.515691    5074 generic.go:356] "Generic (PLEG): container finished" podID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerID="2881ea54d8db861c9c0bbb5996ebec69aaf3cde559ee697a9883a864aa9f9a92" exitCode=60
Nov 04 00:15:18.516188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.515973    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerDied","Data":"2881ea54d8db861c9c0bbb5996ebec69aaf3cde559ee697a9883a864aa9f9a92"}
Nov 04 00:15:18.516390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.516364    5074 scope.go:122] "RemoveContainer" containerID="2881ea54d8db861c9c0bbb5996ebec69aaf3cde559ee697a9883a864aa9f9a92"
Nov 04 00:15:18.516704 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:18.516678    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw_odh-ai-gateway-infra(a9f28e54-550c-4e0f-b8a3-3a0fef812ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5"
Nov 04 00:15:18.528137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.528111    5074 generic.go:356] "Generic (PLEG): container finished" podID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerID="bad6e5565b8b2c887316445a6e62a1b33c0d8513632d9f8172d7126c13f97e3b" exitCode=60
Nov 04 00:15:18.528323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.528164    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" event={"ID":"5030efe2-f18e-4640-88f2-c65df1984cfc","Type":"ContainerDied","Data":"bad6e5565b8b2c887316445a6e62a1b33c0d8513632d9f8172d7126c13f97e3b"}
Nov 04 00:15:18.528862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.528842    5074 scope.go:122] "RemoveContainer" containerID="bad6e5565b8b2c887316445a6e62a1b33c0d8513632d9f8172d7126c13f97e3b"
Nov 04 00:15:18.529231 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:18.529197    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm_odh-ai-gateway-infra(5030efe2-f18e-4640-88f2-c65df1984cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc"
Nov 04 00:15:18.537174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.537149    5074 scope.go:122] "RemoveContainer" containerID="533d750334639e5e3346cb631bf4df6b26f1ee701363d12ff16f0d035fb01c0b"
Nov 04 00:15:18.537433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.537412    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerID="f0850131b7d28cf70d1dc8457c658b6a0b7ca19771b3483acf4153f24ef8bcaa" exitCode=60
Nov 04 00:15:18.537507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.537478    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" event={"ID":"4d713377-aee5-467f-a7c2-c9131a33f61e","Type":"ContainerDied","Data":"f0850131b7d28cf70d1dc8457c658b6a0b7ca19771b3483acf4153f24ef8bcaa"}
Nov 04 00:15:18.537913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.537894    5074 scope.go:122] "RemoveContainer" containerID="f0850131b7d28cf70d1dc8457c658b6a0b7ca19771b3483acf4153f24ef8bcaa"
Nov 04 00:15:18.538282 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:18.538258    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4_odh-ai-gateway-infra(4d713377-aee5-467f-a7c2-c9131a33f61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e"
Nov 04 00:15:18.558159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.554955    5074 scope.go:122] "RemoveContainer" containerID="4056cac76b6f318804afb43ae64ca8151af500d8401ce0e3e68f5e2f296f586f"
Nov 04 00:15:18.576861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:18.576834    5074 scope.go:122] "RemoveContainer" containerID="966c0396e1b1b16e05fac56771aeafffc1d374960783575f984e652dc3d23955"
Nov 04 00:15:19.010726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.010682    5074 scope.go:122] "RemoveContainer" containerID="08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc"
Nov 04 00:15:19.010966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.010889    5074 scope.go:122] "RemoveContainer" containerID="99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66"
Nov 04 00:15:19.011341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.011036    5074 scope.go:122] "RemoveContainer" containerID="ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e"
Nov 04 00:15:19.011341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.011146    5074 scope.go:122] "RemoveContainer" containerID="3aff2efb0fde425f87454eda4a38b7316fb2597ae489d16e219acc3fea4228f1"
Nov 04 00:15:19.544266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.544230    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"]
Nov 04 00:15:19.555354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.555323    5074 generic.go:356] "Generic (PLEG): container finished" podID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerID="cb6d934179bce9acf753d3f981e6ff7d2b2d73937a50600f52e67f2c95934b9b" exitCode=60
Nov 04 00:15:19.555530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.555391    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" event={"ID":"4d1fb445-cd3b-465f-bf6e-5a098acb44b7","Type":"ContainerDied","Data":"cb6d934179bce9acf753d3f981e6ff7d2b2d73937a50600f52e67f2c95934b9b"}
Nov 04 00:15:19.555530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.555445    5074 scope.go:122] "RemoveContainer" containerID="8d4804b55cc521b6b4714eb0badcf84bcd2ccedfcba46bc11d32f7e3349f95a1"
Nov 04 00:15:19.555978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.555949    5074 scope.go:122] "RemoveContainer" containerID="cb6d934179bce9acf753d3f981e6ff7d2b2d73937a50600f52e67f2c95934b9b"
Nov 04 00:15:19.556343 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:19.556317    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j_odh-ai-gateway-infra(4d1fb445-cd3b-465f-bf6e-5a098acb44b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7"
Nov 04 00:15:19.560504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.560481    5074 generic.go:356] "Generic (PLEG): container finished" podID="656ec7be-d09e-4148-9159-21e2be885498" containerID="816a803560e61be1f59296f6392f32af607299ee8384550dc59d08a58c3ade20" exitCode=60
Nov 04 00:15:19.560612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.560595    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" event={"ID":"656ec7be-d09e-4148-9159-21e2be885498","Type":"ContainerDied","Data":"816a803560e61be1f59296f6392f32af607299ee8384550dc59d08a58c3ade20"}
Nov 04 00:15:19.561052 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.561034    5074 scope.go:122] "RemoveContainer" containerID="816a803560e61be1f59296f6392f32af607299ee8384550dc59d08a58c3ade20"
Nov 04 00:15:19.561456 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:19.561410    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd_odh-ai-gateway-infra(656ec7be-d09e-4148-9159-21e2be885498)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" podUID="656ec7be-d09e-4148-9159-21e2be885498"
Nov 04 00:15:19.562727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.562709    5074 generic.go:356] "Generic (PLEG): container finished" podID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerID="e3d829ca179f0b1626322b1c931dccc6b574f0e09c1735866197eb3be2f75d10" exitCode=60
Nov 04 00:15:19.562814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.562800    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" event={"ID":"ac2af7b7-02d9-419d-803a-051f2e66ed0f","Type":"ContainerDied","Data":"e3d829ca179f0b1626322b1c931dccc6b574f0e09c1735866197eb3be2f75d10"}
Nov 04 00:15:19.563145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.563124    5074 scope.go:122] "RemoveContainer" containerID="e3d829ca179f0b1626322b1c931dccc6b574f0e09c1735866197eb3be2f75d10"
Nov 04 00:15:19.563408 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:19.563389    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf_odh-ai-gateway-infra(ac2af7b7-02d9-419d-803a-051f2e66ed0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f"
Nov 04 00:15:19.564712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.564688    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"]
Nov 04 00:15:19.565294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.565277    5074 generic.go:356] "Generic (PLEG): container finished" podID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerID="659352bea3bf4c37a260320ed913b5ba884010a5da209b58e7d254d04c8e1702" exitCode=60
Nov 04 00:15:19.565369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.565342    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerDied","Data":"659352bea3bf4c37a260320ed913b5ba884010a5da209b58e7d254d04c8e1702"}
Nov 04 00:15:19.565694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.565673    5074 scope.go:122] "RemoveContainer" containerID="659352bea3bf4c37a260320ed913b5ba884010a5da209b58e7d254d04c8e1702"
Nov 04 00:15:19.566070 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:19.566046    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895855-r8h7s_odh-ai-gateway-infra(aff3587a-a841-4d1a-9151-e7045abe8c7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b"
Nov 04 00:15:19.571749 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.571713    5074 generic.go:356] "Generic (PLEG): container finished" podID="f10b6447-61da-42f7-b173-474779526489" containerID="3cf1b821e757a58df6a0f67cf9ee7a4925c98f5455de1ba86bb9deaf09bc912d" exitCode=60
Nov 04 00:15:19.571836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.571766    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" event={"ID":"f10b6447-61da-42f7-b173-474779526489","Type":"ContainerDied","Data":"3cf1b821e757a58df6a0f67cf9ee7a4925c98f5455de1ba86bb9deaf09bc912d"}
Nov 04 00:15:19.572108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.572093    5074 scope.go:122] "RemoveContainer" containerID="3cf1b821e757a58df6a0f67cf9ee7a4925c98f5455de1ba86bb9deaf09bc912d"
Nov 04 00:15:19.572378 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:19.572350    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn_odh-ai-gateway-infra(f10b6447-61da-42f7-b173-474779526489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" podUID="f10b6447-61da-42f7-b173-474779526489"
Nov 04 00:15:19.574823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.574760    5074 generic.go:356] "Generic (PLEG): container finished" podID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerID="3f1cb9abb6504511a8fc89e9a45f6592d9283836ae0c2d941c1ae62ec522cc9e" exitCode=60
Nov 04 00:15:19.574930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.574873    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" event={"ID":"dedc8c18-693b-4c80-a315-abdf48d07bc3","Type":"ContainerDied","Data":"3f1cb9abb6504511a8fc89e9a45f6592d9283836ae0c2d941c1ae62ec522cc9e"}
Nov 04 00:15:19.575217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.575199    5074 scope.go:122] "RemoveContainer" containerID="3f1cb9abb6504511a8fc89e9a45f6592d9283836ae0c2d941c1ae62ec522cc9e"
Nov 04 00:15:19.575482 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:19.575461    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l_odh-ai-gateway-infra(dedc8c18-693b-4c80-a315-abdf48d07bc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3"
Nov 04 00:15:19.595159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.595105    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"]
Nov 04 00:15:19.612791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.612505    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"]
Nov 04 00:15:19.755145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.754952    5074 scope.go:122] "RemoveContainer" containerID="ef3b79d6c68d1ed18d449d4190d41b5e7cafba037ba7b7763a0821407198af0a"
Nov 04 00:15:19.781420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.781388    5074 scope.go:122] "RemoveContainer" containerID="c231350a8ff2c133544259b3b1b06388031fd996b4e9ae8c72d7cc6d63a4cfd2"
Nov 04 00:15:19.817483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.817459    5074 scope.go:122] "RemoveContainer" containerID="b511a58fd85147234c7983220db861c7d5a339d0fcb9e67efd239a93105544ec"
Nov 04 00:15:19.842670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.842594    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:19.852197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.852176    5074 scope.go:122] "RemoveContainer" containerID="44df65d7360ec6f543a7ddf1751c1395f94e205682bd4d0e5437c57180ea056f"
Nov 04 00:15:19.897367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.894755    5074 scope.go:122] "RemoveContainer" containerID="95ea4444aaf45f317f0a8d4af5296f84383ddefe0dfb97f15386ee0252e31916"
Nov 04 00:15:19.961357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.961328    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:19.962166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.962137    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6\" (UniqueName: \"kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6\") pod \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") "
Nov 04 00:15:19.962541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.962518    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca\") pod \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\" (UID: \"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb\") "
Nov 04 00:15:19.963639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.963613    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca" pod "6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" (UID: "6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:19.967246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.967223    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6" pod "6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" (UID: "6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb"). InnerVolumeSpecName "kube-api-access-nhsj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:19.970369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.970342    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:19.974795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:19.974776    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:20.013929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.010511    5074 scope.go:122] "RemoveContainer" containerID="9dbd1e160217b12aba87fed94720be0ee12933fa81934d55c234e62e9e832757"
Nov 04 00:15:20.013929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.010713    5074 scope.go:122] "RemoveContainer" containerID="686c40f2707eb22d9d544f954ed62b6639c3cd67db660b009c110108288eaa87"
Nov 04 00:15:20.013929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.011383    5074 scope.go:122] "RemoveContainer" containerID="cfe217984155f1147f93c624d87075a934b845d16275369955c0043613d51df3"
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.064231    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca\") pod \"4d713377-aee5-467f-a7c2-c9131a33f61e\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") "
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.064345    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca\") pod \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") "
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.064648    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca" pod "4d713377-aee5-467f-a7c2-c9131a33f61e" (UID: "4d713377-aee5-467f-a7c2-c9131a33f61e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.064685    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca" pod "a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" (UID: "a9f28e54-550c-4e0f-b8a3-3a0fef812ae5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.064928    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7\" (UniqueName: \"kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7\") pod \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\" (UID: \"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5\") "
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.064992    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8\" (UniqueName: \"kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8\") pod \"4d713377-aee5-467f-a7c2-c9131a33f61e\" (UID: \"4d713377-aee5-467f-a7c2-c9131a33f61e\") "
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.065586    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.065605    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d713377-aee5-467f-a7c2-c9131a33f61e-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.065619    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.065807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.065634    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsj6\" (UniqueName: \"kubernetes.io/projected/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb-kube-api-access-nhsj6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.072806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.068960    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7" pod "a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" (UID: "a9f28e54-550c-4e0f-b8a3-3a0fef812ae5"). InnerVolumeSpecName "kube-api-access-44js7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:20.072806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.069599    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8" pod "4d713377-aee5-467f-a7c2-c9131a33f61e" (UID: "4d713377-aee5-467f-a7c2-c9131a33f61e"). InnerVolumeSpecName "kube-api-access-6mps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:20.166467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.166442    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca\") pod \"5030efe2-f18e-4640-88f2-c65df1984cfc\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") "
Nov 04 00:15:20.166590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.166571    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc\" (UniqueName: \"kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc\") pod \"5030efe2-f18e-4640-88f2-c65df1984cfc\" (UID: \"5030efe2-f18e-4640-88f2-c65df1984cfc\") "
Nov 04 00:15:20.166857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.166836    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca" pod "5030efe2-f18e-4640-88f2-c65df1984cfc" (UID: "5030efe2-f18e-4640-88f2-c65df1984cfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:20.167647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.167439    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5030efe2-f18e-4640-88f2-c65df1984cfc-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.167647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.167464    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44js7\" (UniqueName: \"kubernetes.io/projected/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5-kube-api-access-44js7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.167647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.167476    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mps8\" (UniqueName: \"kubernetes.io/projected/4d713377-aee5-467f-a7c2-c9131a33f61e-kube-api-access-6mps8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.202779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.202271    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc" pod "5030efe2-f18e-4640-88f2-c65df1984cfc" (UID: "5030efe2-f18e-4640-88f2-c65df1984cfc"). InnerVolumeSpecName "kube-api-access-7nrbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:20.269090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.269055    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrbc\" (UniqueName: \"kubernetes.io/projected/5030efe2-f18e-4640-88f2-c65df1984cfc-kube-api-access-7nrbc\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.587631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.587591    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerStarted","Data":"b46eb0fcbde0cc52caf5529031a52aa378f77188982f8eb3f799178ec0c543fd"}
Nov 04 00:15:20.589746 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.589712    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerStarted","Data":"f768c185ffa71b28ef8c162903d9c00f8de0fd000723168312980dc89dfb1987"}
Nov 04 00:15:20.591826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.591793    5074 generic.go:356] "Generic (PLEG): container finished" podID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerID="b0fc6f8308957c6ca4f72a8918dc35b19989544e59acceb0bc5b10b5fd0665d7" exitCode=60
Nov 04 00:15:20.591951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.591818    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" event={"ID":"c43a2226-93f4-46a6-94ca-db9ec3552ac0","Type":"ContainerDied","Data":"b0fc6f8308957c6ca4f72a8918dc35b19989544e59acceb0bc5b10b5fd0665d7"}
Nov 04 00:15:20.591951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.591882    5074 scope.go:122] "RemoveContainer" containerID="3aff2efb0fde425f87454eda4a38b7316fb2597ae489d16e219acc3fea4228f1"
Nov 04 00:15:20.592200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.592179    5074 scope.go:122] "RemoveContainer" containerID="b0fc6f8308957c6ca4f72a8918dc35b19989544e59acceb0bc5b10b5fd0665d7"
Nov 04 00:15:20.592419 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:20.592400    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6_odh-ai-gateway-infra(c43a2226-93f4-46a6-94ca-db9ec3552ac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0"
Nov 04 00:15:20.593815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.593794    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"]
Nov 04 00:15:20.594441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.594423    5074 generic.go:356] "Generic (PLEG): container finished" podID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerID="2e7782bda683ec382ae8ffe8fafc9735df07e79d017594673426e12900f8a4a1" exitCode=60
Nov 04 00:15:20.594520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.594499    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" event={"ID":"862b2732-ead5-4f3a-9b68-881210dcb7e9","Type":"ContainerDied","Data":"2e7782bda683ec382ae8ffe8fafc9735df07e79d017594673426e12900f8a4a1"}
Nov 04 00:15:20.594802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.594787    5074 scope.go:122] "RemoveContainer" containerID="2e7782bda683ec382ae8ffe8fafc9735df07e79d017594673426e12900f8a4a1"
Nov 04 00:15:20.594993 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:20.594972    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797_odh-ai-gateway-infra(862b2732-ead5-4f3a-9b68-881210dcb7e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9"
Nov 04 00:15:20.595908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.595883    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm" event={"ID":"5030efe2-f18e-4640-88f2-c65df1984cfc","Type":"ContainerDied","Data":"4ee2b1febb107f3e60666154da2b2531e47025ce52b45c8486a6aa307873a62b"}
Nov 04 00:15:20.596003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.595889    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"
Nov 04 00:15:20.608884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.608854    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4" event={"ID":"4d713377-aee5-467f-a7c2-c9131a33f61e","Type":"ContainerDied","Data":"51a64e06ccb72003ac6a7c26f8611a11f125898bb53f75ec5c175b6bb31bdb3a"}
Nov 04 00:15:20.609086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.609002    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"
Nov 04 00:15:20.616139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.616116    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"]
Nov 04 00:15:20.639584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.639468    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"
Nov 04 00:15:20.639584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.639516    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p" event={"ID":"6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb","Type":"ContainerDied","Data":"73e336dd6646e0fe6dda515a08f1ef9acb211594843781542f1a492bd6287e3f"}
Nov 04 00:15:20.644867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.644838    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw" event={"ID":"a9f28e54-550c-4e0f-b8a3-3a0fef812ae5","Type":"ContainerDied","Data":"5515cf57d783295bfbf59a0bfc2cad5d2dc2a29c6eecf6699180fd5665fb4416"}
Nov 04 00:15:20.644975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.644959    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"
Nov 04 00:15:20.661651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.661590    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"]
Nov 04 00:15:20.721704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.721672    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"]
Nov 04 00:15:20.805184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.805145    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"]
Nov 04 00:15:20.822624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.822384    5074 scope.go:122] "RemoveContainer" containerID="08e241f77e0f8f127ffc2cef5460eb37314ffbf66a57ae110910d677330acccc"
Nov 04 00:15:20.874124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.874089    5074 scope.go:122] "RemoveContainer" containerID="bad6e5565b8b2c887316445a6e62a1b33c0d8513632d9f8172d7126c13f97e3b"
Nov 04 00:15:20.874533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.874092    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"]
Nov 04 00:15:20.895727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.895696    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"]
Nov 04 00:15:20.909853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.908305    5074 scope.go:122] "RemoveContainer" containerID="f0850131b7d28cf70d1dc8457c658b6a0b7ca19771b3483acf4153f24ef8bcaa"
Nov 04 00:15:20.922784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.922489    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:20.934919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.934842    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895855-nxm5p"]
Nov 04 00:15:20.955235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.955206    5074 scope.go:122] "RemoveContainer" containerID="c2d585471d79faa3e8a40d16e47102da4618966d03ee593063c4ce14f78b6344"
Nov 04 00:15:20.955526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.955494    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"]
Nov 04 00:15:20.964188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.964166    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895855-z45k4"]
Nov 04 00:15:20.972344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.972293    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"]
Nov 04 00:15:20.983079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.983026    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895855-2lxhw"]
Nov 04 00:15:20.983318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:20.983298    5074 scope.go:122] "RemoveContainer" containerID="2881ea54d8db861c9c0bbb5996ebec69aaf3cde559ee697a9883a864aa9f9a92"
Nov 04 00:15:21.012270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.012236    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk\" (UniqueName: \"kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk\") pod \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") "
Nov 04 00:15:21.013438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.012515    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca\") pod \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\" (UID: \"4d1fb445-cd3b-465f-bf6e-5a098acb44b7\") "
Nov 04 00:15:21.014043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.013551    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca" pod "4d1fb445-cd3b-465f-bf6e-5a098acb44b7" (UID: "4d1fb445-cd3b-465f-bf6e-5a098acb44b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:21.014043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.013580    5074 scope.go:122] "RemoveContainer" containerID="78de06fcb09d7f24615a15fdde675b7ba254bc1c648ca5ddf0bfd46ce17a5cfd"
Nov 04 00:15:21.069578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.069525    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk" pod "4d1fb445-cd3b-465f-bf6e-5a098acb44b7" (UID: "4d1fb445-cd3b-465f-bf6e-5a098acb44b7"). InnerVolumeSpecName "kube-api-access-9g5gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:21.085014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.084982    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" path="/var/lib/kubelet/pods/4d713377-aee5-467f-a7c2-c9131a33f61e/volumes"
Nov 04 00:15:21.085873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.085853    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" path="/var/lib/kubelet/pods/6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb/volumes"
Nov 04 00:15:21.086566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.086547    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" path="/var/lib/kubelet/pods/a9f28e54-550c-4e0f-b8a3-3a0fef812ae5/volumes"
Nov 04 00:15:21.087809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.087791    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"]
Nov 04 00:15:21.087942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.087932    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895855-x52sm"]
Nov 04 00:15:21.088059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.088049    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"]
Nov 04 00:15:21.088145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.088137    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8365e2ea-fkc2x"]
Nov 04 00:15:21.118159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.114620    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5gk\" (UniqueName: \"kubernetes.io/projected/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-kube-api-access-9g5gk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.118159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.114654    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1fb445-cd3b-465f-bf6e-5a098acb44b7-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.193691    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:21.225799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.222256    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:21.239123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.239089    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:21.253825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.253000    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:21.322039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.321717    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca\") pod \"f10b6447-61da-42f7-b173-474779526489\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") "
Nov 04 00:15:21.322039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.321926    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca\") pod \"656ec7be-d09e-4148-9159-21e2be885498\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") "
Nov 04 00:15:21.322039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.321974    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk\" (UniqueName: \"kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk\") pod \"656ec7be-d09e-4148-9159-21e2be885498\" (UID: \"656ec7be-d09e-4148-9159-21e2be885498\") "
Nov 04 00:15:21.322322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.322180    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk\" (UniqueName: \"kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk\") pod \"f10b6447-61da-42f7-b173-474779526489\" (UID: \"f10b6447-61da-42f7-b173-474779526489\") "
Nov 04 00:15:21.336363 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.336106    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca" pod "f10b6447-61da-42f7-b173-474779526489" (UID: "f10b6447-61da-42f7-b173-474779526489"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:21.336816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.336693    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca" pod "656ec7be-d09e-4148-9159-21e2be885498" (UID: "656ec7be-d09e-4148-9159-21e2be885498"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:21.358240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.358173    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk" pod "656ec7be-d09e-4148-9159-21e2be885498" (UID: "656ec7be-d09e-4148-9159-21e2be885498"). InnerVolumeSpecName "kube-api-access-vwvsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:21.360349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.360317    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk" pod "f10b6447-61da-42f7-b173-474779526489" (UID: "f10b6447-61da-42f7-b173-474779526489"). InnerVolumeSpecName "kube-api-access-ql6qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:21.427235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.427205    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46\" (UniqueName: \"kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46\") pod \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") "
Nov 04 00:15:21.427393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.427245    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca\") pod \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\" (UID: \"ac2af7b7-02d9-419d-803a-051f2e66ed0f\") "
Nov 04 00:15:21.427471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.427398    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca\") pod \"aff3587a-a841-4d1a-9151-e7045abe8c7b\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") "
Nov 04 00:15:21.427471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.427443    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr\" (UniqueName: \"kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr\") pod \"aff3587a-a841-4d1a-9151-e7045abe8c7b\" (UID: \"aff3587a-a841-4d1a-9151-e7045abe8c7b\") "
Nov 04 00:15:21.427726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.427696    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca" pod "ac2af7b7-02d9-419d-803a-051f2e66ed0f" (UID: "ac2af7b7-02d9-419d-803a-051f2e66ed0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:21.427946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.427799    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca" pod "aff3587a-a841-4d1a-9151-e7045abe8c7b" (UID: "aff3587a-a841-4d1a-9151-e7045abe8c7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:21.428136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.428114    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2af7b7-02d9-419d-803a-051f2e66ed0f-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.428189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.428144    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656ec7be-d09e-4148-9159-21e2be885498-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.428189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.428159    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvsk\" (UniqueName: \"kubernetes.io/projected/656ec7be-d09e-4148-9159-21e2be885498-kube-api-access-vwvsk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.428189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.428175    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6qk\" (UniqueName: \"kubernetes.io/projected/f10b6447-61da-42f7-b173-474779526489-kube-api-access-ql6qk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.428286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.428189    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff3587a-a841-4d1a-9151-e7045abe8c7b-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.428286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.428203    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10b6447-61da-42f7-b173-474779526489-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.429674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.429654    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr" pod "aff3587a-a841-4d1a-9151-e7045abe8c7b" (UID: "aff3587a-a841-4d1a-9151-e7045abe8c7b"). InnerVolumeSpecName "kube-api-access-mrtpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:21.430017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.429990    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46" pod "ac2af7b7-02d9-419d-803a-051f2e66ed0f" (UID: "ac2af7b7-02d9-419d-803a-051f2e66ed0f"). InnerVolumeSpecName "kube-api-access-4tk46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:21.447448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.447382    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:21.530704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.530660    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tk46\" (UniqueName: \"kubernetes.io/projected/ac2af7b7-02d9-419d-803a-051f2e66ed0f-kube-api-access-4tk46\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.530866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.530707    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrtpr\" (UniqueName: \"kubernetes.io/projected/aff3587a-a841-4d1a-9151-e7045abe8c7b-kube-api-access-mrtpr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.629831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.629791    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"]
Nov 04 00:15:21.632092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.632062    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj\" (UniqueName: \"kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj\") pod \"dedc8c18-693b-4c80-a315-abdf48d07bc3\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") "
Nov 04 00:15:21.632262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.632159    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca\") pod \"dedc8c18-693b-4c80-a315-abdf48d07bc3\" (UID: \"dedc8c18-693b-4c80-a315-abdf48d07bc3\") "
Nov 04 00:15:21.632495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.632472    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca" pod "dedc8c18-693b-4c80-a315-abdf48d07bc3" (UID: "dedc8c18-693b-4c80-a315-abdf48d07bc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:21.632838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.632818    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dedc8c18-693b-4c80-a315-abdf48d07bc3-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.634501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.634468    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj" pod "dedc8c18-693b-4c80-a315-abdf48d07bc3" (UID: "dedc8c18-693b-4c80-a315-abdf48d07bc3"). InnerVolumeSpecName "kube-api-access-qw2zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:21.660619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.660594    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"
Nov 04 00:15:21.660810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.660586    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn" event={"ID":"f10b6447-61da-42f7-b173-474779526489","Type":"ContainerDied","Data":"039cbd5d60d5fdfbca6d3310c33fdf66992e2d93bf334112570e1ebcc166663a"}
Nov 04 00:15:21.660810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.660728    5074 scope.go:122] "RemoveContainer" containerID="3cf1b821e757a58df6a0f67cf9ee7a4925c98f5455de1ba86bb9deaf09bc912d"
Nov 04 00:15:21.662263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.662238    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"
Nov 04 00:15:21.662364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.662266    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l" event={"ID":"dedc8c18-693b-4c80-a315-abdf48d07bc3","Type":"ContainerDied","Data":"fdf90b0285be53904b825981a884f9adadccabd408034d806a1ccc1325be2c0b"}
Nov 04 00:15:21.665047 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.665018    5074 generic.go:356] "Generic (PLEG): container finished" podID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerID="b46eb0fcbde0cc52caf5529031a52aa378f77188982f8eb3f799178ec0c543fd" exitCode=60
Nov 04 00:15:21.665183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.665106    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerDied","Data":"b46eb0fcbde0cc52caf5529031a52aa378f77188982f8eb3f799178ec0c543fd"}
Nov 04 00:15:21.666830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.665488    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"]
Nov 04 00:15:21.668380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.668290    5074 generic.go:356] "Generic (PLEG): container finished" podID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerID="f768c185ffa71b28ef8c162903d9c00f8de0fd000723168312980dc89dfb1987" exitCode=60
Nov 04 00:15:21.668380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.668365    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerDied","Data":"f768c185ffa71b28ef8c162903d9c00f8de0fd000723168312980dc89dfb1987"}
Nov 04 00:15:21.674793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.674731    5074 generic.go:356] "Generic (PLEG): container finished" podID="216a9882-650f-4a5d-96b5-414b34d235ac" containerID="346695e858ba957868ecbe27daead4ede98db854167a9a1d401d4d7282afe09a" exitCode=60
Nov 04 00:15:21.675031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.674792    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" event={"ID":"216a9882-650f-4a5d-96b5-414b34d235ac","Type":"ContainerDied","Data":"346695e858ba957868ecbe27daead4ede98db854167a9a1d401d4d7282afe09a"}
Nov 04 00:15:21.676013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.675528    5074 scope.go:122] "RemoveContainer" containerID="346695e858ba957868ecbe27daead4ede98db854167a9a1d401d4d7282afe09a"
Nov 04 00:15:21.676013 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:21.675890    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf_odh-ai-gateway-infra(216a9882-650f-4a5d-96b5-414b34d235ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" podUID="216a9882-650f-4a5d-96b5-414b34d235ac"
Nov 04 00:15:21.678896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.678864    5074 scope.go:122] "RemoveContainer" containerID="3f1cb9abb6504511a8fc89e9a45f6592d9283836ae0c2d941c1ae62ec522cc9e"
Nov 04 00:15:21.684856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.684828    5074 generic.go:356] "Generic (PLEG): container finished" podID="870046cf-6a49-4993-a0d4-6941cfea5636" containerID="cacf1bee8624edbd862ef30efb443f5565548e741e9798ec605d440cfe895dff" exitCode=60
Nov 04 00:15:21.684950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.684872    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" event={"ID":"870046cf-6a49-4993-a0d4-6941cfea5636","Type":"ContainerDied","Data":"cacf1bee8624edbd862ef30efb443f5565548e741e9798ec605d440cfe895dff"}
Nov 04 00:15:21.685369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.685344    5074 scope.go:122] "RemoveContainer" containerID="cacf1bee8624edbd862ef30efb443f5565548e741e9798ec605d440cfe895dff"
Nov 04 00:15:21.685753 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:21.685707    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2_odh-ai-gateway-infra(870046cf-6a49-4993-a0d4-6941cfea5636)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" podUID="870046cf-6a49-4993-a0d4-6941cfea5636"
Nov 04 00:15:21.688347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.688145    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"]
Nov 04 00:15:21.692508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.691733    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j" event={"ID":"4d1fb445-cd3b-465f-bf6e-5a098acb44b7","Type":"ContainerDied","Data":"da49dfdeaef41472a014a151f50ceb2278e26a0d0f4b3d56718980a861460984"}
Nov 04 00:15:21.692508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.691883    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"
Nov 04 00:15:21.695331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.695113    5074 generic.go:356] "Generic (PLEG): container finished" podID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerID="59451e50fbc74c2d54ed0868956c2e7ec1c6960e7968815275234c90ff483e0c" exitCode=60
Nov 04 00:15:21.695331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.695198    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" event={"ID":"b3190980-0a5b-40e0-9961-b3f16873b79f","Type":"ContainerDied","Data":"59451e50fbc74c2d54ed0868956c2e7ec1c6960e7968815275234c90ff483e0c"}
Nov 04 00:15:21.698870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.696874    5074 scope.go:122] "RemoveContainer" containerID="59451e50fbc74c2d54ed0868956c2e7ec1c6960e7968815275234c90ff483e0c"
Nov 04 00:15:21.698870 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:21.697200    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298_odh-ai-gateway-infra(b3190980-0a5b-40e0-9961-b3f16873b79f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f"
Nov 04 00:15:21.705591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.703062    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd" event={"ID":"656ec7be-d09e-4148-9159-21e2be885498","Type":"ContainerDied","Data":"720be0825fc9d56a078d838d0a8066ca1bfa20e3fabdab4ba059873fda4a2885"}
Nov 04 00:15:21.705591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.703175    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"
Nov 04 00:15:21.717756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.714180    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf" event={"ID":"ac2af7b7-02d9-419d-803a-051f2e66ed0f","Type":"ContainerDied","Data":"a965590ffd9b3a48b8c1cbf7f7f11f113cc721e6c88e9d67c045b9b2bcf2f9ef"}
Nov 04 00:15:21.717756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.714297    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"
Nov 04 00:15:21.722539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.719652    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s" event={"ID":"aff3587a-a841-4d1a-9151-e7045abe8c7b","Type":"ContainerDied","Data":"cce830475e005fd933118990acd7d6b432a18dfe5c93cafc5a9d30e5e294ef67"}
Nov 04 00:15:21.722539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.719781    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"
Nov 04 00:15:21.735695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.735665    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw2zj\" (UniqueName: \"kubernetes.io/projected/dedc8c18-693b-4c80-a315-abdf48d07bc3-kube-api-access-qw2zj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:21.805503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.804566    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"]
Nov 04 00:15:21.851215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.820735    5074 scope.go:122] "RemoveContainer" containerID="ed300744bee5c8d49f6f2754fa757e879de15dddfc92555ab67fb7c6b555b91e"
Nov 04 00:15:21.862100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.862044    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"]
Nov 04 00:15:21.869978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.869942    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895855-dbwwn"]
Nov 04 00:15:21.876095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.876070    5074 scope.go:122] "RemoveContainer" containerID="99b75b9a4d278c2dc9412b49f9a0621855571eca84e15c2d3ae044c97282ff66"
Nov 04 00:15:21.899476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.899451    5074 scope.go:122] "RemoveContainer" containerID="cfe217984155f1147f93c624d87075a934b845d16275369955c0043613d51df3"
Nov 04 00:15:21.936611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.936583    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"]
Nov 04 00:15:21.941623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.941597    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895855-q5g5l"]
Nov 04 00:15:21.943612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.943591    5074 scope.go:122] "RemoveContainer" containerID="686c40f2707eb22d9d544f954ed62b6639c3cd67db660b009c110108288eaa87"
Nov 04 00:15:21.949080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.948995    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:21.955519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.955360    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"]
Nov 04 00:15:21.960280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.960252    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895855-2cpcd"]
Nov 04 00:15:21.964550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.964530    5074 scope.go:122] "RemoveContainer" containerID="cb6d934179bce9acf753d3f981e6ff7d2b2d73937a50600f52e67f2c95934b9b"
Nov 04 00:15:21.982610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.982403    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"]
Nov 04 00:15:21.983223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.983103    5074 scope.go:122] "RemoveContainer" containerID="9dbd1e160217b12aba87fed94720be0ee12933fa81934d55c234e62e9e832757"
Nov 04 00:15:21.990478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:21.990431    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895855-r8h7s"]
Nov 04 00:15:22.013700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.010198    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"]
Nov 04 00:15:22.013700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.013319    5074 scope.go:122] "RemoveContainer" containerID="816a803560e61be1f59296f6392f32af607299ee8384550dc59d08a58c3ade20"
Nov 04 00:15:22.032486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.032172    5074 scope.go:122] "RemoveContainer" containerID="e3d829ca179f0b1626322b1c931dccc6b574f0e09c1735866197eb3be2f75d10"
Nov 04 00:15:22.034351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.034282    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895855-vjlcf"]
Nov 04 00:15:22.057379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.053346    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58\" (UniqueName: \"kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58\") pod \"093d3a77-e009-4e3d-aa72-3a7488d8e295\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") "
Nov 04 00:15:22.057379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.053594    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca\") pod \"093d3a77-e009-4e3d-aa72-3a7488d8e295\" (UID: \"093d3a77-e009-4e3d-aa72-3a7488d8e295\") "
Nov 04 00:15:22.057379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.054621    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca" pod "093d3a77-e009-4e3d-aa72-3a7488d8e295" (UID: "093d3a77-e009-4e3d-aa72-3a7488d8e295"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:22.060027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.059561    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58" pod "093d3a77-e009-4e3d-aa72-3a7488d8e295" (UID: "093d3a77-e009-4e3d-aa72-3a7488d8e295"). InnerVolumeSpecName "kube-api-access-h2f58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:22.060027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.059950    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"]
Nov 04 00:15:22.067691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.067664    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895855-n655j"]
Nov 04 00:15:22.156933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.156882    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d3a77-e009-4e3d-aa72-3a7488d8e295-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.156933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.156910    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2f58\" (UniqueName: \"kubernetes.io/projected/093d3a77-e009-4e3d-aa72-3a7488d8e295-kube-api-access-h2f58\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.170444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.170417    5074 scope.go:122] "RemoveContainer" containerID="659352bea3bf4c37a260320ed913b5ba884010a5da209b58e7d254d04c8e1702"
Nov 04 00:15:22.173622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.173272    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:22.183180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.183162    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:22.187493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.187470    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:22.359825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.359729    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca\") pod \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") "
Nov 04 00:15:22.359825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.359785    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca\") pod \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") "
Nov 04 00:15:22.359825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.359814    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca\") pod \"862b2732-ead5-4f3a-9b68-881210dcb7e9\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") "
Nov 04 00:15:22.360128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.359873    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c\" (UniqueName: \"kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c\") pod \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\" (UID: \"c43a2226-93f4-46a6-94ca-db9ec3552ac0\") "
Nov 04 00:15:22.360128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.359911    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl\" (UniqueName: \"kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl\") pod \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\" (UID: \"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f\") "
Nov 04 00:15:22.360128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.359994    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7\" (UniqueName: \"kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7\") pod \"862b2732-ead5-4f3a-9b68-881210dcb7e9\" (UID: \"862b2732-ead5-4f3a-9b68-881210dcb7e9\") "
Nov 04 00:15:22.360289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.360072    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca" pod "c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" (UID: "c330931a-3ea9-4fe6-85fd-0a1d13c4a20f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:22.360289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.360165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca" pod "c43a2226-93f4-46a6-94ca-db9ec3552ac0" (UID: "c43a2226-93f4-46a6-94ca-db9ec3552ac0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:22.360289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.360225    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca" pod "862b2732-ead5-4f3a-9b68-881210dcb7e9" (UID: "862b2732-ead5-4f3a-9b68-881210dcb7e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:22.360628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.360607    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.360628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.360627    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a2226-93f4-46a6-94ca-db9ec3552ac0-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.360852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.360637    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862b2732-ead5-4f3a-9b68-881210dcb7e9-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.362117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.362091    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c" pod "c43a2226-93f4-46a6-94ca-db9ec3552ac0" (UID: "c43a2226-93f4-46a6-94ca-db9ec3552ac0"). InnerVolumeSpecName "kube-api-access-pg89c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:22.362210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.362160    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7" pod "862b2732-ead5-4f3a-9b68-881210dcb7e9" (UID: "862b2732-ead5-4f3a-9b68-881210dcb7e9"). InnerVolumeSpecName "kube-api-access-fvfx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:22.362210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.362177    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl" pod "c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" (UID: "c330931a-3ea9-4fe6-85fd-0a1d13c4a20f"). InnerVolumeSpecName "kube-api-access-9mzbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:22.462240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.462206    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg89c\" (UniqueName: \"kubernetes.io/projected/c43a2226-93f4-46a6-94ca-db9ec3552ac0-kube-api-access-pg89c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.462240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.462237    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzbl\" (UniqueName: \"kubernetes.io/projected/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f-kube-api-access-9mzbl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.462240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.462248    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfx7\" (UniqueName: \"kubernetes.io/projected/862b2732-ead5-4f3a-9b68-881210dcb7e9-kube-api-access-fvfx7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:22.762671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.762639    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"
Nov 04 00:15:22.762891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.762639    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg" event={"ID":"093d3a77-e009-4e3d-aa72-3a7488d8e295","Type":"ContainerDied","Data":"6952fe2a7e364c1901ac5242311e7ca80defd389c9dfe7b1fed3307e6f76e786"}
Nov 04 00:15:22.762891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.762789    5074 scope.go:122] "RemoveContainer" containerID="b46eb0fcbde0cc52caf5529031a52aa378f77188982f8eb3f799178ec0c543fd"
Nov 04 00:15:22.764126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.764100    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"
Nov 04 00:15:22.764284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.764107    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2" event={"ID":"c330931a-3ea9-4fe6-85fd-0a1d13c4a20f","Type":"ContainerDied","Data":"6b3d6e39803a1a229c872c5c2ac431de839eed281396086241819effc2947087"}
Nov 04 00:15:22.766173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.766147    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"
Nov 04 00:15:22.766296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.766176    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6" event={"ID":"c43a2226-93f4-46a6-94ca-db9ec3552ac0","Type":"ContainerDied","Data":"8848894a208ea3c5c17ed43021d013a20d0959ffe422360c8e9de54eeffbe09b"}
Nov 04 00:15:22.773671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.773655    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"
Nov 04 00:15:22.773810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.773685    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797" event={"ID":"862b2732-ead5-4f3a-9b68-881210dcb7e9","Type":"ContainerDied","Data":"dfc3b5631742a5254b23240a30f9bf0176e49680e5e5dc50eb0fb69da630bc07"}
Nov 04 00:15:22.782518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.782496    5074 scope.go:122] "RemoveContainer" containerID="f768c185ffa71b28ef8c162903d9c00f8de0fd000723168312980dc89dfb1987"
Nov 04 00:15:22.783629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.783607    5074 generic.go:356] "Generic (PLEG): container finished" podID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerID="826fdde9ea0c9fae01a4ddd359dcc22d52531c2ed11ff4ce39fc4c6f40f34dda" exitCode=60
Nov 04 00:15:22.783719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.783667    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" event={"ID":"222b3305-bcdd-4f33-a787-7ff020d19e19","Type":"ContainerDied","Data":"826fdde9ea0c9fae01a4ddd359dcc22d52531c2ed11ff4ce39fc4c6f40f34dda"}
Nov 04 00:15:22.784089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.784069    5074 scope.go:122] "RemoveContainer" containerID="826fdde9ea0c9fae01a4ddd359dcc22d52531c2ed11ff4ce39fc4c6f40f34dda"
Nov 04 00:15:22.784370 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:22.784350    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx_odh-ai-gateway-infra(222b3305-bcdd-4f33-a787-7ff020d19e19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19"
Nov 04 00:15:22.792311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.792292    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"]
Nov 04 00:15:22.801233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.801210    5074 scope.go:122] "RemoveContainer" containerID="b0fc6f8308957c6ca4f72a8918dc35b19989544e59acceb0bc5b10b5fd0665d7"
Nov 04 00:15:22.837819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.837792    5074 scope.go:122] "RemoveContainer" containerID="2e7782bda683ec382ae8ffe8fafc9735df07e79d017594673426e12900f8a4a1"
Nov 04 00:15:22.855785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.855758    5074 scope.go:122] "RemoveContainer" containerID="78de06fcb09d7f24615a15fdde675b7ba254bc1c648ca5ddf0bfd46ce17a5cfd"
Nov 04 00:15:22.859060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.859034    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"]
Nov 04 00:15:22.863663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.863643    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895855-sgnt2"]
Nov 04 00:15:22.869970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.869945    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"]
Nov 04 00:15:22.875253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.875231    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895855-mh797"]
Nov 04 00:15:22.893804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.887904    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"]
Nov 04 00:15:22.894088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.894064    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"]
Nov 04 00:15:22.908895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.908867    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895855-br9l6"]
Nov 04 00:15:22.924814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.924482    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"]
Nov 04 00:15:22.944477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.941627    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"]
Nov 04 00:15:22.944477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.941682    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895855-grdjg"]
Nov 04 00:15:22.997571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:22.997548    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:23.040056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.039971    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" path="/var/lib/kubelet/pods/093d3a77-e009-4e3d-aa72-3a7488d8e295/volumes"
Nov 04 00:15:23.041369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.041345    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b0ecb58-3fdb-4562-bc76-c5dd2e481f24" path="/var/lib/kubelet/pods/4b0ecb58-3fdb-4562-bc76-c5dd2e481f24/volumes"
Nov 04 00:15:23.042586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.042563    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" path="/var/lib/kubelet/pods/4d1fb445-cd3b-465f-bf6e-5a098acb44b7/volumes"
Nov 04 00:15:23.044632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.044610    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" path="/var/lib/kubelet/pods/5030efe2-f18e-4640-88f2-c65df1984cfc/volumes"
Nov 04 00:15:23.045379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.045358    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656ec7be-d09e-4148-9159-21e2be885498" path="/var/lib/kubelet/pods/656ec7be-d09e-4148-9159-21e2be885498/volumes"
Nov 04 00:15:23.046301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.046283    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" path="/var/lib/kubelet/pods/862b2732-ead5-4f3a-9b68-881210dcb7e9/volumes"
Nov 04 00:15:23.047409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.047393    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" path="/var/lib/kubelet/pods/ac2af7b7-02d9-419d-803a-051f2e66ed0f/volumes"
Nov 04 00:15:23.047958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.047944    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" path="/var/lib/kubelet/pods/aff3587a-a841-4d1a-9151-e7045abe8c7b/volumes"
Nov 04 00:15:23.048455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.048442    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" path="/var/lib/kubelet/pods/c330931a-3ea9-4fe6-85fd-0a1d13c4a20f/volumes"
Nov 04 00:15:23.049301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.049283    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" path="/var/lib/kubelet/pods/c43a2226-93f4-46a6-94ca-db9ec3552ac0/volumes"
Nov 04 00:15:23.050129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.050109    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" path="/var/lib/kubelet/pods/dedc8c18-693b-4c80-a315-abdf48d07bc3/volumes"
Nov 04 00:15:23.050566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.050547    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10b6447-61da-42f7-b173-474779526489" path="/var/lib/kubelet/pods/f10b6447-61da-42f7-b173-474779526489/volumes"
Nov 04 00:15:23.070321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.070292    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp\" (UniqueName: \"kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp\") pod \"b3190980-0a5b-40e0-9961-b3f16873b79f\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") "
Nov 04 00:15:23.070541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.070515    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca\") pod \"b3190980-0a5b-40e0-9961-b3f16873b79f\" (UID: \"b3190980-0a5b-40e0-9961-b3f16873b79f\") "
Nov 04 00:15:23.070963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.070942    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca" pod "b3190980-0a5b-40e0-9961-b3f16873b79f" (UID: "b3190980-0a5b-40e0-9961-b3f16873b79f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:23.071286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.071264    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3190980-0a5b-40e0-9961-b3f16873b79f-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:23.073971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.073949    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp" pod "b3190980-0a5b-40e0-9961-b3f16873b79f" (UID: "b3190980-0a5b-40e0-9961-b3f16873b79f"). InnerVolumeSpecName "kube-api-access-k72sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:23.139993 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.139960    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:23.173191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.171772    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca\") pod \"216a9882-650f-4a5d-96b5-414b34d235ac\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") "
Nov 04 00:15:23.173191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.171999    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs\" (UniqueName: \"kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs\") pod \"216a9882-650f-4a5d-96b5-414b34d235ac\" (UID: \"216a9882-650f-4a5d-96b5-414b34d235ac\") "
Nov 04 00:15:23.173191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.172470    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca" pod "216a9882-650f-4a5d-96b5-414b34d235ac" (UID: "216a9882-650f-4a5d-96b5-414b34d235ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:23.173191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.172675    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k72sp\" (UniqueName: \"kubernetes.io/projected/b3190980-0a5b-40e0-9961-b3f16873b79f-kube-api-access-k72sp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:23.173191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.172697    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a9882-650f-4a5d-96b5-414b34d235ac-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:23.174480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.174456    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs" pod "216a9882-650f-4a5d-96b5-414b34d235ac" (UID: "216a9882-650f-4a5d-96b5-414b34d235ac"). InnerVolumeSpecName "kube-api-access-jszzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:23.231624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.231593    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:23.273783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.273371    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca\") pod \"870046cf-6a49-4993-a0d4-6941cfea5636\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") "
Nov 04 00:15:23.273783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.273422    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk\" (UniqueName: \"kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk\") pod \"870046cf-6a49-4993-a0d4-6941cfea5636\" (UID: \"870046cf-6a49-4993-a0d4-6941cfea5636\") "
Nov 04 00:15:23.274048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.274024    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszzs\" (UniqueName: \"kubernetes.io/projected/216a9882-650f-4a5d-96b5-414b34d235ac-kube-api-access-jszzs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:23.274094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.274081    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca" pod "870046cf-6a49-4993-a0d4-6941cfea5636" (UID: "870046cf-6a49-4993-a0d4-6941cfea5636"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:23.275856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.275828    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk" pod "870046cf-6a49-4993-a0d4-6941cfea5636" (UID: "870046cf-6a49-4993-a0d4-6941cfea5636"). InnerVolumeSpecName "kube-api-access-qntzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:23.375597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.375488    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870046cf-6a49-4993-a0d4-6941cfea5636-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:23.375597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.375533    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntzk\" (UniqueName: \"kubernetes.io/projected/870046cf-6a49-4993-a0d4-6941cfea5636-kube-api-access-qntzk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:23.800876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.800843    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"
Nov 04 00:15:23.801086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.800880    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298" event={"ID":"b3190980-0a5b-40e0-9961-b3f16873b79f","Type":"ContainerDied","Data":"d42fcb61a911f667cb56b8f7a37650ef33f34351c4f15d34c697098ef449bb19"}
Nov 04 00:15:23.801086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.800960    5074 scope.go:122] "RemoveContainer" containerID="59451e50fbc74c2d54ed0868956c2e7ec1c6960e7968815275234c90ff483e0c"
Nov 04 00:15:23.807823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.807790    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf" event={"ID":"216a9882-650f-4a5d-96b5-414b34d235ac","Type":"ContainerDied","Data":"54d0b155503b832b28627dbc495f527e48a5d50acf1f01ff2da7460b72a50611"}
Nov 04 00:15:23.807823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.807810    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"
Nov 04 00:15:23.816857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.816753    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2" event={"ID":"870046cf-6a49-4993-a0d4-6941cfea5636","Type":"ContainerDied","Data":"0ca75e3bb81afcd6f0d26eaecdd533914cf7188d1e58db8ebab29e0f27e0c166"}
Nov 04 00:15:23.816942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.816912    5074 scope.go:122] "RemoveContainer" containerID="346695e858ba957868ecbe27daead4ede98db854167a9a1d401d4d7282afe09a"
Nov 04 00:15:23.816942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.816928    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"
Nov 04 00:15:23.827602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.827575    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"]
Nov 04 00:15:23.834024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.833992    5074 scope.go:122] "RemoveContainer" containerID="cacf1bee8624edbd862ef30efb443f5565548e741e9798ec605d440cfe895dff"
Nov 04 00:15:23.843433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.843317    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"]
Nov 04 00:15:23.854995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.854969    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895855-sw298"]
Nov 04 00:15:23.886328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.886294    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"]
Nov 04 00:15:23.892833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.892803    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895855-v7zqf"]
Nov 04 00:15:23.928485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.928448    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"]
Nov 04 00:15:23.934125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.934095    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895855-4cts2"]
Nov 04 00:15:23.990262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:23.990237    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:24.084501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.084469    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg\" (UniqueName: \"kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg\") pod \"222b3305-bcdd-4f33-a787-7ff020d19e19\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") "
Nov 04 00:15:24.084950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.084751    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca\") pod \"222b3305-bcdd-4f33-a787-7ff020d19e19\" (UID: \"222b3305-bcdd-4f33-a787-7ff020d19e19\") "
Nov 04 00:15:24.085067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.085042    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca" pod "222b3305-bcdd-4f33-a787-7ff020d19e19" (UID: "222b3305-bcdd-4f33-a787-7ff020d19e19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:24.085391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.085372    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222b3305-bcdd-4f33-a787-7ff020d19e19-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:24.086878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.086856    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg" pod "222b3305-bcdd-4f33-a787-7ff020d19e19" (UID: "222b3305-bcdd-4f33-a787-7ff020d19e19"). InnerVolumeSpecName "kube-api-access-qp5hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:24.187218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.187154    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5hg\" (UniqueName: \"kubernetes.io/projected/222b3305-bcdd-4f33-a787-7ff020d19e19-kube-api-access-qp5hg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:24.881999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.881958    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx" event={"ID":"222b3305-bcdd-4f33-a787-7ff020d19e19","Type":"ContainerDied","Data":"d939a75cf74e7e1ba9eff6600568c337a1c0c31090e2dd385f277ff651525134"}
Nov 04 00:15:24.882281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.882259    5074 scope.go:122] "RemoveContainer" containerID="826fdde9ea0c9fae01a4ddd359dcc22d52531c2ed11ff4ce39fc4c6f40f34dda"
Nov 04 00:15:24.882487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.882470    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"
Nov 04 00:15:24.982853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.982821    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"]
Nov 04 00:15:24.998257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:24.998220    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895855-zwzkx"]
Nov 04 00:15:25.189518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:25.188661    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" path="/var/lib/kubelet/pods/216a9882-650f-4a5d-96b5-414b34d235ac/volumes"
Nov 04 00:15:25.189518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:25.189394    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" path="/var/lib/kubelet/pods/222b3305-bcdd-4f33-a787-7ff020d19e19/volumes"
Nov 04 00:15:25.190155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:25.190056    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" path="/var/lib/kubelet/pods/870046cf-6a49-4993-a0d4-6941cfea5636/volumes"
Nov 04 00:15:25.191140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:25.191115    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" path="/var/lib/kubelet/pods/b3190980-0a5b-40e0-9961-b3f16873b79f/volumes"
Nov 04 00:15:25.481763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:25.481639    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:15:25.482306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:25.482280    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:15:28.025766 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:28.025712    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:15:28.874555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.874521    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"]
Nov 04 00:15:28.875356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875337    5074 cpu_manager.go:414] "containerMap: removing container" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.875356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875354    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875366    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875371    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875381    5074 cpu_manager.go:414] "containerMap: removing container" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875386    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875393    5074 cpu_manager.go:414] "containerMap: removing container" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875399    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875406    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875410    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875420    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875425    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875432    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875437    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875444    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875448    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875453    5074 cpu_manager.go:414] "containerMap: removing container" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875458    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875467    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875471    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875479    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875483    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875490    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875495    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875501    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:28.875501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875506    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875514    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875518    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875524    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875528    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875533    5074 cpu_manager.go:414] "containerMap: removing container" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875538    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875544    5074 cpu_manager.go:414] "containerMap: removing container" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875548    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875561    5074 cpu_manager.go:414] "containerMap: removing container" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875565    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875573    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875577    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875583    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875587    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875594    5074 cpu_manager.go:414] "containerMap: removing container" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875598    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875605    5074 cpu_manager.go:414] "containerMap: removing container" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875610    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875616    5074 cpu_manager.go:414] "containerMap: removing container" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875621    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875628    5074 cpu_manager.go:414] "containerMap: removing container" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875632    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875640    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875644    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875651    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875656    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875663    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875667    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875685    5074 cpu_manager.go:414] "containerMap: removing container" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875690    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875697    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875701    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875709    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875713    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875719    5074 cpu_manager.go:414] "containerMap: removing container" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875724    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875731    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875746    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875756    5074 cpu_manager.go:414] "containerMap: removing container" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875760    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875767    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875771    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875779    5074 cpu_manager.go:414] "containerMap: removing container" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875783    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875805    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875811    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875820    5074 cpu_manager.go:414] "containerMap: removing container" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875825    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875831    5074 cpu_manager.go:414] "containerMap: removing container" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875836    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875845    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875850    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875857    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875862    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875868    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875872    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875880    5074 cpu_manager.go:414] "containerMap: removing container" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875884    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875892    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875897    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875907    5074 cpu_manager.go:414] "containerMap: removing container" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875912    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875920    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875924    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875932    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875937    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875945    5074 cpu_manager.go:414] "containerMap: removing container" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.875950    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876111    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876126    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876134    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876143    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876152    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876160    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876167    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876177    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876185    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876191    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876197    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876206    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876218    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876227    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876235    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876244    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876252    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876260    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876267    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876274    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.876242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876284    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876293    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876302    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876310    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876318    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876328    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876337    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3190980-0a5b-40e0-9961-b3f16873b79f" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876344    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10b6447-61da-42f7-b173-474779526489" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876353    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876361    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876368    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876376    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876387    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876396    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876407    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876416    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a2226-93f4-46a6-94ca-db9ec3552ac0" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876426    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="222b3305-bcdd-4f33-a787-7ff020d19e19" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876434    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876447    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876456    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876464    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedc8c18-693b-4c80-a315-abdf48d07bc3" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876476    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876486    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="216a9882-650f-4a5d-96b5-414b34d235ac" containerName="cleanup"
Nov 04 00:15:28.878911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.876495    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:28.882796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.882727    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.885818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.885793    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"]
Nov 04 00:15:28.886422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.886398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.886526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.886486    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfjq\" (UniqueName: \"kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.886776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.886715    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.887192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.887171    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:15:28.889041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.889025    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:15:28.989477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.988308    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.989477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.988373    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfjq\" (UniqueName: \"kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.989477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.988532    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.989477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.989368    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:28.999786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:28.992896    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:29.035094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.035061    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfjq\" (UniqueName: \"kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq\") pod \"maas-api-revoke-sub-f677d1a6-5qqhw\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:29.044219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.044193    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"]
Nov 04 00:15:29.044356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.044249    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769e5e62-2mrhd"]
Nov 04 00:15:29.216624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.213295    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:29.274869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.274833    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"]
Nov 04 00:15:29.296990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.276303    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:29.408174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408145    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408181    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408187    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408196    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408201    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408211    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408216    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c330931a-3ea9-4fe6-85fd-0a1d13c4a20f" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408240    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408247    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408268    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408273    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d713377-aee5-467f-a7c2-c9131a33f61e" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408293    5074 cpu_manager.go:414] "containerMap: removing container" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:29.408315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408297    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408482    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1fb445-cd3b-465f-bf6e-5a098acb44b7" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408493    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="862b2732-ead5-4f3a-9b68-881210dcb7e9" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408501    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2af7b7-02d9-419d-803a-051f2e66ed0f" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408507    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5030efe2-f18e-4640-88f2-c65df1984cfc" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408513    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9ac2f8-eb23-41c8-a397-1a47a4d7d4bb" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408518    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff3587a-a841-4d1a-9151-e7045abe8c7b" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408525    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="656ec7be-d09e-4148-9159-21e2be885498" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408532    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="093d3a77-e009-4e3d-aa72-3a7488d8e295" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408540    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="870046cf-6a49-4993-a0d4-6941cfea5636" containerName="cleanup"
Nov 04 00:15:29.409074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.408547    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f28e54-550c-4e0f-b8a3-3a0fef812ae5" containerName="cleanup"
Nov 04 00:15:29.434761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.420408    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"]
Nov 04 00:15:29.439759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.420532    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.458376 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.447714    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"payload-processing-external-model-e2e--4ce224e7-dockercfg-bl9ld\""
Nov 04 00:15:29.458376 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.447811    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fa0d0a\"/\"agc-cred-0397af4da7f2\""
Nov 04 00:15:29.531585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.531006    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlx9\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.531585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.531109    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.531585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.531359    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.531585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.531428    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.712017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.711132    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.712017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.711223    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.712017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.711272    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlx9\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.712017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.711351    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.723909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.713088    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.723909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.713704    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.736521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.736454    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlx9\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.802075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.801377    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") " pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:29.841971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:29.841930    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"]
Nov 04 00:15:30.074606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.074528    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:30.102205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.102171    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw" event={"ID":"4e2e61f6-765a-454d-a78b-02ef55d71942","Type":"ContainerStarted","Data":"35ed72a8ef199a41bf5bb2d5c65e7be23fe6b2f2c2a17af45082d37a3df2e5c0"}
Nov 04 00:15:30.378167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.378133    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"]
Nov 04 00:15:30.379132 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:30.379101    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5eec4d_752f_4c85_bfb5_da2843072f25.slice/crio-31a90c3b51011a7d1c8df16570d92bfa9fc8d193b67b675704455320ac4fdf4f WatchSource:0}: Error finding container 31a90c3b51011a7d1c8df16570d92bfa9fc8d193b67b675704455320ac4fdf4f: Status 404 returned error can't find the container with id 31a90c3b51011a7d1c8df16570d92bfa9fc8d193b67b675704455320ac4fdf4f
Nov 04 00:15:30.440474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.440443    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"]
Nov 04 00:15:30.446828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.446806    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.451447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.451425    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-426f93ec944b\""
Nov 04 00:15:30.452936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.452911    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"]
Nov 04 00:15:30.526833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.526802    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.527016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.526862    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.527016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.526986    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.527143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.527028    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcglb\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.628371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.628277    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.628371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.628337    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.628603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.628387    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.628603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.628416    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcglb\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.629168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.629137    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.629321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.629241    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.631107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.631081    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.640902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.640880    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcglb\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:30.784251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:30.784217    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:31.024190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.024156    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc56a90-6844-40fe-9674-8c3e5a05b498" path="/var/lib/kubelet/pods/fcc56a90-6844-40fe-9674-8c3e5a05b498/volumes"
Nov 04 00:15:31.084689 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:31.084656    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1af856_4e6a_4e98_b738_1887e89620ae.slice/crio-4d1754ec1fc05ef23783a7288a07ee94d2068cac96e0ced4941c48fb0d6ea37b WatchSource:0}: Error finding container 4d1754ec1fc05ef23783a7288a07ee94d2068cac96e0ced4941c48fb0d6ea37b: Status 404 returned error can't find the container with id 4d1754ec1fc05ef23783a7288a07ee94d2068cac96e0ced4941c48fb0d6ea37b
Nov 04 00:15:31.085038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.085004    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"]
Nov 04 00:15:31.115791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.115724    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" event={"ID":"ff1af856-4e6a-4e98-b738-1887e89620ae","Type":"ContainerStarted","Data":"4d1754ec1fc05ef23783a7288a07ee94d2068cac96e0ced4941c48fb0d6ea37b"}
Nov 04 00:15:31.118201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.118135    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" event={"ID":"7c5eec4d-752f-4c85-bfb5-da2843072f25","Type":"ContainerStarted","Data":"00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167"}
Nov 04 00:15:31.118313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.118277    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" event={"ID":"7c5eec4d-752f-4c85-bfb5-da2843072f25","Type":"ContainerStarted","Data":"31a90c3b51011a7d1c8df16570d92bfa9fc8d193b67b675704455320ac4fdf4f"}
Nov 04 00:15:31.118373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.118348    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:31.120240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.120213    5074 generic.go:356] "Generic (PLEG): container finished" podID="4e2e61f6-765a-454d-a78b-02ef55d71942" containerID="b167296645c11a6e8d71618f31c9bec905dbe28dd5923982d4e76da97bae85d9" exitCode=0
Nov 04 00:15:31.120345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.120279    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw" event={"ID":"4e2e61f6-765a-454d-a78b-02ef55d71942","Type":"ContainerDied","Data":"b167296645c11a6e8d71618f31c9bec905dbe28dd5923982d4e76da97bae85d9"}
Nov 04 00:15:31.138692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:31.138650    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" podStartSLOduration=2.138637806 podStartE2EDuration="2.138637806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:31.138069342 +0000 UTC m=+2796.237669486" watchObservedRunningTime="2026-11-04 00:15:31.138637806 +0000 UTC m=+2796.238237950"
Nov 04 00:15:32.173470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.166963    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" event={"ID":"ff1af856-4e6a-4e98-b738-1887e89620ae","Type":"ContainerStarted","Data":"e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2"}
Nov 04 00:15:32.173470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.169041    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:32.194481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.194405    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" podStartSLOduration=2.194389433 podStartE2EDuration="2.194389433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:32.194164889 +0000 UTC m=+2797.293765033" watchObservedRunningTime="2026-11-04 00:15:32.194389433 +0000 UTC m=+2797.293989577"
Nov 04 00:15:32.260183 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.260149    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2697a0/agc-cred-426f93ec944b: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:32.260417 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.260396    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:32.260597 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.260575    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials podName:ff1af856-4e6a-4e98-b738-1887e89620ae nodeName:}" failed. No retries permitted until 2026-11-04 00:15:32.760554261 +0000 UTC m=+2797.860154386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae") : secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:32.260707 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.260591    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fa0d0a/agc-cred-0397af4da7f2: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:32.260816 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.260804    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:32.260961 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.260949    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials podName:7c5eec4d-752f-4c85-bfb5-da2843072f25 nodeName:}" failed. No retries permitted until 2026-11-04 00:15:32.760932111 +0000 UTC m=+2797.860532256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25") : secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:32.271771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.271724    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"]
Nov 04 00:15:32.631575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.631542    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:32.670273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.670233    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"]
Nov 04 00:15:32.769978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.769941    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq\" (UniqueName: \"kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq\") pod \"4e2e61f6-765a-454d-a78b-02ef55d71942\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") "
Nov 04 00:15:32.770171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.770152    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca\") pod \"4e2e61f6-765a-454d-a78b-02ef55d71942\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") "
Nov 04 00:15:32.770227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.770203    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca\") pod \"4e2e61f6-765a-454d-a78b-02ef55d71942\" (UID: \"4e2e61f6-765a-454d-a78b-02ef55d71942\") "
Nov 04 00:15:32.770656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.770601    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca" pod "4e2e61f6-765a-454d-a78b-02ef55d71942" (UID: "4e2e61f6-765a-454d-a78b-02ef55d71942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:32.770823 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.770709    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2697a0/agc-cred-426f93ec944b: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:32.770823 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.770733    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:32.770823 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.770805    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials podName:ff1af856-4e6a-4e98-b738-1887e89620ae nodeName:}" failed. No retries permitted until 2026-11-04 00:15:33.770786394 +0000 UTC m=+2798.870386531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae") : secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:32.771091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.771068    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:32.771302 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.771271    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fa0d0a/agc-cred-0397af4da7f2: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:32.771445 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.771302    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:32.771445 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:32.771371    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials podName:7c5eec4d-752f-4c85-bfb5-da2843072f25 nodeName:}" failed. No retries permitted until 2026-11-04 00:15:33.771354053 +0000 UTC m=+2798.870954177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25") : secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:32.772571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.772544    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq" pod "4e2e61f6-765a-454d-a78b-02ef55d71942" (UID: "4e2e61f6-765a-454d-a78b-02ef55d71942"). InnerVolumeSpecName "kube-api-access-rcfjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:32.772723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.772702    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca" pod "4e2e61f6-765a-454d-a78b-02ef55d71942" (UID: "4e2e61f6-765a-454d-a78b-02ef55d71942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:15:32.804955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.804923    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"]
Nov 04 00:15:32.806315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.806285    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4e2e61f6-765a-454d-a78b-02ef55d71942" containerName="revoke-keys"
Nov 04 00:15:32.806315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.806312    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2e61f6-765a-454d-a78b-02ef55d71942" containerName="revoke-keys"
Nov 04 00:15:32.806540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.806525    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2e61f6-765a-454d-a78b-02ef55d71942" containerName="revoke-keys"
Nov 04 00:15:32.814570 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.814115    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.818131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.818059    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"]
Nov 04 00:15:32.873064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.872593    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kzw\" (UniqueName: \"kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.873064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.872649    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.873064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.872958    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.873374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.873262    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfjq\" (UniqueName: \"kubernetes.io/projected/4e2e61f6-765a-454d-a78b-02ef55d71942-kube-api-access-rcfjq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:32.873374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.873282    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2e61f6-765a-454d-a78b-02ef55d71942-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:32.974617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.974572    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kzw\" (UniqueName: \"kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.974617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.974619    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.974903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.974711    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.975606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.975580    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.977362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.977337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:32.987582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:32.987556    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kzw\" (UniqueName: \"kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw\") pod \"maas-api-revoke-sub-e4820179-nmldl\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:33.152943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:33.152871    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:33.177138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:33.177093    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"
Nov 04 00:15:33.177577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:33.177092    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw" event={"ID":"4e2e61f6-765a-454d-a78b-02ef55d71942","Type":"ContainerDied","Data":"35ed72a8ef199a41bf5bb2d5c65e7be23fe6b2f2c2a17af45082d37a3df2e5c0"}
Nov 04 00:15:33.177577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:33.177216    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ed72a8ef199a41bf5bb2d5c65e7be23fe6b2f2c2a17af45082d37a3df2e5c0"
Nov 04 00:15:33.444347 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:33.444311    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49061992_7aea_4bd6_a7e8_5ba4b5f4cf25.slice/crio-2f18a43776b0b192ac91a5df51a89a50db0d54327b15a3f5ad197015704002b0 WatchSource:0}: Error finding container 2f18a43776b0b192ac91a5df51a89a50db0d54327b15a3f5ad197015704002b0: Status 404 returned error can't find the container with id 2f18a43776b0b192ac91a5df51a89a50db0d54327b15a3f5ad197015704002b0
Nov 04 00:15:33.445182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:33.445154    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"]
Nov 04 00:15:33.787781 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:33.787689    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fa0d0a/agc-cred-0397af4da7f2: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:33.787781 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:33.787721    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:33.787969 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:33.787785    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials podName:7c5eec4d-752f-4c85-bfb5-da2843072f25 nodeName:}" failed. No retries permitted until 2026-11-04 00:15:35.787771636 +0000 UTC m=+2800.887371779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25") : secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:33.787969 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:33.787794    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2697a0/agc-cred-426f93ec944b: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:33.787969 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:33.787821    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:33.787969 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:33.787873    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials podName:ff1af856-4e6a-4e98-b738-1887e89620ae nodeName:}" failed. No retries permitted until 2026-11-04 00:15:35.787856871 +0000 UTC m=+2800.887457023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae") : secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:34.043645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:34.043258    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"]
Nov 04 00:15:34.048478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:34.048447    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07dc9d0-v7lhf"]
Nov 04 00:15:34.188717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:34.188668    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl" event={"ID":"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25","Type":"ContainerStarted","Data":"75456528775b57caed04c6d7a54456af4fb50d2dab2459bbe29a22f6513e6ee2"}
Nov 04 00:15:34.189190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:34.188749    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl" event={"ID":"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25","Type":"ContainerStarted","Data":"2f18a43776b0b192ac91a5df51a89a50db0d54327b15a3f5ad197015704002b0"}
Nov 04 00:15:34.212692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:34.212642    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl" podStartSLOduration=2.212628724 podStartE2EDuration="2.212628724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:34.207900186 +0000 UTC m=+2799.307500331" watchObservedRunningTime="2026-11-04 00:15:34.212628724 +0000 UTC m=+2799.312228865"
Nov 04 00:15:35.037497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:35.037458    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dddeb40-2e71-456a-8fde-a9379b41a963" path="/var/lib/kubelet/pods/1dddeb40-2e71-456a-8fde-a9379b41a963/volumes"
Nov 04 00:15:35.199488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:35.199452    5074 generic.go:356] "Generic (PLEG): container finished" podID="49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" containerID="75456528775b57caed04c6d7a54456af4fb50d2dab2459bbe29a22f6513e6ee2" exitCode=0
Nov 04 00:15:35.199911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:35.199537    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl" event={"ID":"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25","Type":"ContainerDied","Data":"75456528775b57caed04c6d7a54456af4fb50d2dab2459bbe29a22f6513e6ee2"}
Nov 04 00:15:35.831776 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:35.831721    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2697a0/agc-cred-426f93ec944b: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:35.831776 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:35.831767    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68: secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:35.832177 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:35.831832    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials podName:ff1af856-4e6a-4e98-b738-1887e89620ae nodeName:}" failed. No retries permitted until 2026-11-04 00:15:39.831814683 +0000 UTC m=+2804.931414817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae") : secret "agc-cred-426f93ec944b" not found
Nov 04 00:15:35.832177 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:35.831879    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fa0d0a/agc-cred-0397af4da7f2: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:35.832177 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:35.831909    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp: secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:35.832177 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:35.831967    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials podName:7c5eec4d-752f-4c85-bfb5-da2843072f25 nodeName:}" failed. No retries permitted until 2026-11-04 00:15:39.831951185 +0000 UTC m=+2804.931551322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25") : secret "agc-cred-0397af4da7f2" not found
Nov 04 00:15:36.434912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.434877    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:36.449699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.449666    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca\") pod \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") "
Nov 04 00:15:36.449931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.449913    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca\") pod \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") "
Nov 04 00:15:36.450065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.450039    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca" pod "49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" (UID: "49061992-7aea-4bd6-a7e8-5ba4b5f4cf25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:36.450133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.450107    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw\" (UniqueName: \"kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw\") pod \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\" (UID: \"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25\") "
Nov 04 00:15:36.450684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.450660    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:36.452613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.452582    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca" pod "49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" (UID: "49061992-7aea-4bd6-a7e8-5ba4b5f4cf25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:15:36.452716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.452648    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw" pod "49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" (UID: "49061992-7aea-4bd6-a7e8-5ba4b5f4cf25"). InnerVolumeSpecName "kube-api-access-z4kzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:36.557816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.556935    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:36.557816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:36.556975    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4kzw\" (UniqueName: \"kubernetes.io/projected/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25-kube-api-access-z4kzw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:37.229353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:37.229317    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"
Nov 04 00:15:37.229530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:37.229348    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl" event={"ID":"49061992-7aea-4bd6-a7e8-5ba4b5f4cf25","Type":"ContainerDied","Data":"2f18a43776b0b192ac91a5df51a89a50db0d54327b15a3f5ad197015704002b0"}
Nov 04 00:15:37.229530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:37.229398    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f18a43776b0b192ac91a5df51a89a50db0d54327b15a3f5ad197015704002b0"
Nov 04 00:15:38.177670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.177618    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" podUID="7c5eec4d-752f-4c85-bfb5-da2843072f25" containerName="payload-processing" containerID="cri-o://00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167" gracePeriod=25
Nov 04 00:15:38.484171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.484145    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:38.592938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.592899    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume\") pod \"7c5eec4d-752f-4c85-bfb5-da2843072f25\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") "
Nov 04 00:15:38.593114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.592983    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9\") pod \"7c5eec4d-752f-4c85-bfb5-da2843072f25\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") "
Nov 04 00:15:38.593162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.593131    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume\") pod \"7c5eec4d-752f-4c85-bfb5-da2843072f25\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") "
Nov 04 00:15:38.593240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.593223    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials\") pod \"7c5eec4d-752f-4c85-bfb5-da2843072f25\" (UID: \"7c5eec4d-752f-4c85-bfb5-da2843072f25\") "
Nov 04 00:15:38.593318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.593289    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume" pod "7c5eec4d-752f-4c85-bfb5-da2843072f25" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:38.593513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.593489    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume" pod "7c5eec4d-752f-4c85-bfb5-da2843072f25" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:38.593839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.593821    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:38.593902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.593841    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c5eec4d-752f-4c85-bfb5-da2843072f25-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:38.595264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.595242    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9" pod "7c5eec4d-752f-4c85-bfb5-da2843072f25" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25"). InnerVolumeSpecName "kube-api-access-hdlx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:38.595340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.595305    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials" pod "7c5eec4d-752f-4c85-bfb5-da2843072f25" (UID: "7c5eec4d-752f-4c85-bfb5-da2843072f25"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:38.695087 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.695010    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdlx9\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-kube-api-access-hdlx9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:38.695087 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:38.695040    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c5eec4d-752f-4c85-bfb5-da2843072f25-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:39.189317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.189275    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" podUID="ff1af856-4e6a-4e98-b738-1887e89620ae" containerName="payload-processing" containerID="cri-o://e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2" gracePeriod=25
Nov 04 00:15:39.256964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.256930    5074 generic.go:356] "Generic (PLEG): container finished" podID="7c5eec4d-752f-4c85-bfb5-da2843072f25" containerID="00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167" exitCode=0
Nov 04 00:15:39.257152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.256981    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"
Nov 04 00:15:39.257152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.257006    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" event={"ID":"7c5eec4d-752f-4c85-bfb5-da2843072f25","Type":"ContainerDied","Data":"00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167"}
Nov 04 00:15:39.257152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.257059    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp" event={"ID":"7c5eec4d-752f-4c85-bfb5-da2843072f25","Type":"ContainerDied","Data":"31a90c3b51011a7d1c8df16570d92bfa9fc8d193b67b675704455320ac4fdf4f"}
Nov 04 00:15:39.257152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.257080    5074 scope.go:122] "RemoveContainer" containerID="00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167"
Nov 04 00:15:39.314572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.314480    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"]
Nov 04 00:15:39.323660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.323614    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-685dtmtqp"]
Nov 04 00:15:39.324023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.323968    5074 scope.go:122] "RemoveContainer" containerID="00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167"
Nov 04 00:15:39.324393 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:39.324363    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167\": container with ID starting with 00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167 not found: ID does not exist" containerID="00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167"
Nov 04 00:15:39.324494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.324403    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167"} err="failed to get container status \"00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167\": rpc error: code = NotFound desc = could not find container \"00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167\": container with ID starting with 00aa4d0b8e2ff52d6df3c66c72c61d8395d9730f9a01ca0d2207764b42374167 not found: ID does not exist"
Nov 04 00:15:39.508413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.508375    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:39.617769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.617667    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb\") pod \"ff1af856-4e6a-4e98-b738-1887e89620ae\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") "
Nov 04 00:15:39.617957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.617881    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume\") pod \"ff1af856-4e6a-4e98-b738-1887e89620ae\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") "
Nov 04 00:15:39.618046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.618001    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume\") pod \"ff1af856-4e6a-4e98-b738-1887e89620ae\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") "
Nov 04 00:15:39.618114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.618069    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials\") pod \"ff1af856-4e6a-4e98-b738-1887e89620ae\" (UID: \"ff1af856-4e6a-4e98-b738-1887e89620ae\") "
Nov 04 00:15:39.618255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.618217    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume" pod "ff1af856-4e6a-4e98-b738-1887e89620ae" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:39.618640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.618615    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:39.618640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.618609    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume" pod "ff1af856-4e6a-4e98-b738-1887e89620ae" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:39.620547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.620521    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb" pod "ff1af856-4e6a-4e98-b738-1887e89620ae" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae"). InnerVolumeSpecName "kube-api-access-pcglb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:39.620845 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.620823    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials" pod "ff1af856-4e6a-4e98-b738-1887e89620ae" (UID: "ff1af856-4e6a-4e98-b738-1887e89620ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:39.721141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.721102    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcglb\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-kube-api-access-pcglb\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:39.721141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.721135    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1af856-4e6a-4e98-b738-1887e89620ae-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:39.721141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.721144    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff1af856-4e6a-4e98-b738-1887e89620ae-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:39.856987 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:39.853049    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"]
Nov 04 00:15:40.090763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.087797    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"]
Nov 04 00:15:40.107772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.105642    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb9eb6ee-fsvzx"]
Nov 04 00:15:40.270466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.270425    5074 generic.go:356] "Generic (PLEG): container finished" podID="ff1af856-4e6a-4e98-b738-1887e89620ae" containerID="e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2" exitCode=0
Nov 04 00:15:40.270968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.270478    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"
Nov 04 00:15:40.270968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.270506    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" event={"ID":"ff1af856-4e6a-4e98-b738-1887e89620ae","Type":"ContainerDied","Data":"e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2"}
Nov 04 00:15:40.270968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.270562    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68" event={"ID":"ff1af856-4e6a-4e98-b738-1887e89620ae","Type":"ContainerDied","Data":"4d1754ec1fc05ef23783a7288a07ee94d2068cac96e0ced4941c48fb0d6ea37b"}
Nov 04 00:15:40.270968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.270584    5074 scope.go:122] "RemoveContainer" containerID="e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2"
Nov 04 00:15:40.285481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.285463    5074 scope.go:122] "RemoveContainer" containerID="e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2"
Nov 04 00:15:40.285850 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:40.285806    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2\": container with ID starting with e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2 not found: ID does not exist" containerID="e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2"
Nov 04 00:15:40.285970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.285841    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2"} err="failed to get container status \"e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2\": rpc error: code = NotFound desc = could not find container \"e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2\": container with ID starting with e63cc2fafe438d5b75f32d9ef6050159bbbad311b5f9ecbee2916482e9c0afa2 not found: ID does not exist"
Nov 04 00:15:40.299003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.298968    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"]
Nov 04 00:15:40.305833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.305803    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7459w4f68"]
Nov 04 00:15:40.532932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.532898    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"]
Nov 04 00:15:40.533794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.533769    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ff1af856-4e6a-4e98-b738-1887e89620ae" containerName="payload-processing"
Nov 04 00:15:40.533794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.533787    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1af856-4e6a-4e98-b738-1887e89620ae" containerName="payload-processing"
Nov 04 00:15:40.533914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.533809    5074 cpu_manager.go:414] "containerMap: removing container" podUID="49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" containerName="revoke-keys"
Nov 04 00:15:40.533914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.533815    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" containerName="revoke-keys"
Nov 04 00:15:40.533914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.533837    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7c5eec4d-752f-4c85-bfb5-da2843072f25" containerName="payload-processing"
Nov 04 00:15:40.533914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.533843    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5eec4d-752f-4c85-bfb5-da2843072f25" containerName="payload-processing"
Nov 04 00:15:40.534070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.534054    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" containerName="revoke-keys"
Nov 04 00:15:40.534107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.534083    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5eec4d-752f-4c85-bfb5-da2843072f25" containerName="payload-processing"
Nov 04 00:15:40.534145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.534106    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1af856-4e6a-4e98-b738-1887e89620ae" containerName="payload-processing"
Nov 04 00:15:40.540493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.540466    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.545114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.545088    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"]
Nov 04 00:15:40.561868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.561843    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:15:40.562033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.561947    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:15:40.675180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.675143    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.675394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.675288    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shk4w\" (UniqueName: \"kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.675394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.675375    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.783637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.779805    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.783637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.779945    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shk4w\" (UniqueName: \"kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.783637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.780016    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.783637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.781537    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.787354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.787319    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.793569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.793539    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shk4w\" (UniqueName: \"kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w\") pod \"maas-api-revoke-sub-c13c6a8e-qfvfn\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:40.874843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:40.874804    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:41.024438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:41.024397    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5eec4d-752f-4c85-bfb5-da2843072f25" path="/var/lib/kubelet/pods/7c5eec4d-752f-4c85-bfb5-da2843072f25/volumes"
Nov 04 00:15:41.025182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:41.025154    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf2c351-cb98-4ee5-942f-cc517b9a9511" path="/var/lib/kubelet/pods/8bf2c351-cb98-4ee5-942f-cc517b9a9511/volumes"
Nov 04 00:15:41.025759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:41.025721    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1af856-4e6a-4e98-b738-1887e89620ae" path="/var/lib/kubelet/pods/ff1af856-4e6a-4e98-b738-1887e89620ae/volumes"
Nov 04 00:15:41.165258 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:41.165221    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282d83c3_dd16_4fee_aefe_63e45f2f51f5.slice/crio-d56c4aade0ab221c3f02a99ed3bd623f3747cc31dff4d64a40b8e5c097ba6254 WatchSource:0}: Error finding container d56c4aade0ab221c3f02a99ed3bd623f3747cc31dff4d64a40b8e5c097ba6254: Status 404 returned error can't find the container with id d56c4aade0ab221c3f02a99ed3bd623f3747cc31dff4d64a40b8e5c097ba6254
Nov 04 00:15:41.165796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:41.165771    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"]
Nov 04 00:15:41.284183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:41.284130    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn" event={"ID":"282d83c3-dd16-4fee-aefe-63e45f2f51f5","Type":"ContainerStarted","Data":"d56c4aade0ab221c3f02a99ed3bd623f3747cc31dff4d64a40b8e5c097ba6254"}
Nov 04 00:15:42.297424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:42.297383    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn" event={"ID":"282d83c3-dd16-4fee-aefe-63e45f2f51f5","Type":"ContainerStarted","Data":"a11dbdf76babf89ad19fa77d895be8003ea534fa643908c458a06c2f389450fa"}
Nov 04 00:15:42.329134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:42.323879    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn" podStartSLOduration=2.323859858 podStartE2EDuration="2.323859858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:42.319486917 +0000 UTC m=+2807.419087064" watchObservedRunningTime="2026-11-04 00:15:42.323859858 +0000 UTC m=+2807.423460004"
Nov 04 00:15:43.308514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.308472    5074 generic.go:356] "Generic (PLEG): container finished" podID="282d83c3-dd16-4fee-aefe-63e45f2f51f5" containerID="a11dbdf76babf89ad19fa77d895be8003ea534fa643908c458a06c2f389450fa" exitCode=0
Nov 04 00:15:43.308936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.308554    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn" event={"ID":"282d83c3-dd16-4fee-aefe-63e45f2f51f5","Type":"ContainerDied","Data":"a11dbdf76babf89ad19fa77d895be8003ea534fa643908c458a06c2f389450fa"}
Nov 04 00:15:43.414808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.414772    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"]
Nov 04 00:15:43.420587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.420559    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.429820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.429777    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"]
Nov 04 00:15:43.522313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.522274    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.522522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.522502    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.522670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.522656    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drncn\" (UniqueName: \"kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.624169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.624050    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.624379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.624171    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drncn\" (UniqueName: \"kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.624379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.624243    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.624969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.624940    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.626761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.626719    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.636811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.636791    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drncn\" (UniqueName: \"kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn\") pod \"maas-api-revoke-sub-144b2efc-rbdvn\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:43.762607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:43.762574    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:44.051791 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:15:44.051759    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc123ea33_cf7c_41eb_9eca_70fd8119c4ec.slice/crio-998d0286442f00fabffec2639bec64efebd35a7c6f8058b3bd8d45997e7f2968 WatchSource:0}: Error finding container 998d0286442f00fabffec2639bec64efebd35a7c6f8058b3bd8d45997e7f2968: Status 404 returned error can't find the container with id 998d0286442f00fabffec2639bec64efebd35a7c6f8058b3bd8d45997e7f2968
Nov 04 00:15:44.052155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.052130    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"]
Nov 04 00:15:44.319226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.319137    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn" event={"ID":"c123ea33-cf7c-41eb-9eca-70fd8119c4ec","Type":"ContainerStarted","Data":"998d0286442f00fabffec2639bec64efebd35a7c6f8058b3bd8d45997e7f2968"}
Nov 04 00:15:44.479181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.479144    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:44.536872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.536842    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca\") pod \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") "
Nov 04 00:15:44.537018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.536992    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w\" (UniqueName: \"kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w\") pod \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") "
Nov 04 00:15:44.537093 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.537031    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca\") pod \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\" (UID: \"282d83c3-dd16-4fee-aefe-63e45f2f51f5\") "
Nov 04 00:15:44.537487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.537461    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca" pod "282d83c3-dd16-4fee-aefe-63e45f2f51f5" (UID: "282d83c3-dd16-4fee-aefe-63e45f2f51f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:44.537614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.537591    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:44.538964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.538942    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca" pod "282d83c3-dd16-4fee-aefe-63e45f2f51f5" (UID: "282d83c3-dd16-4fee-aefe-63e45f2f51f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:15:44.539569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.539552    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w" pod "282d83c3-dd16-4fee-aefe-63e45f2f51f5" (UID: "282d83c3-dd16-4fee-aefe-63e45f2f51f5"). InnerVolumeSpecName "kube-api-access-shk4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:44.638800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.638713    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shk4w\" (UniqueName: \"kubernetes.io/projected/282d83c3-dd16-4fee-aefe-63e45f2f51f5-kube-api-access-shk4w\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:44.638800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.638784    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d83c3-dd16-4fee-aefe-63e45f2f51f5-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:44.854395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:44.854356    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th" podUID="e4ee1001-32b6-46bc-9329-9a49589e28e7" containerName="payload-processing" containerID="cri-o://00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f" gracePeriod=25
Nov 04 00:15:45.191924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.191895    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:15:45.245363 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.245275    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk\") pod \"e4ee1001-32b6-46bc-9329-9a49589e28e7\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") "
Nov 04 00:15:45.245543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.245498    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume\") pod \"e4ee1001-32b6-46bc-9329-9a49589e28e7\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") "
Nov 04 00:15:45.245613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.245563    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials\") pod \"e4ee1001-32b6-46bc-9329-9a49589e28e7\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") "
Nov 04 00:15:45.245613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.245592    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume\") pod \"e4ee1001-32b6-46bc-9329-9a49589e28e7\" (UID: \"e4ee1001-32b6-46bc-9329-9a49589e28e7\") "
Nov 04 00:15:45.246358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.246313    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume" pod "e4ee1001-32b6-46bc-9329-9a49589e28e7" (UID: "e4ee1001-32b6-46bc-9329-9a49589e28e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:45.246514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.246341    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume" pod "e4ee1001-32b6-46bc-9329-9a49589e28e7" (UID: "e4ee1001-32b6-46bc-9329-9a49589e28e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:45.248341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.248310    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials" pod "e4ee1001-32b6-46bc-9329-9a49589e28e7" (UID: "e4ee1001-32b6-46bc-9329-9a49589e28e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:45.248449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.248429    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk" pod "e4ee1001-32b6-46bc-9329-9a49589e28e7" (UID: "e4ee1001-32b6-46bc-9329-9a49589e28e7"). InnerVolumeSpecName "kube-api-access-j4fpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:45.329797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.329765    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"
Nov 04 00:15:45.329797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.329777    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn" event={"ID":"282d83c3-dd16-4fee-aefe-63e45f2f51f5","Type":"ContainerDied","Data":"d56c4aade0ab221c3f02a99ed3bd623f3747cc31dff4d64a40b8e5c097ba6254"}
Nov 04 00:15:45.330319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.329851    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d56c4aade0ab221c3f02a99ed3bd623f3747cc31dff4d64a40b8e5c097ba6254"
Nov 04 00:15:45.331710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.331684    5074 generic.go:356] "Generic (PLEG): container finished" podID="c123ea33-cf7c-41eb-9eca-70fd8119c4ec" containerID="8d2d212610a2e06e834878103dd3a27ddba2cbfa175947bd8417b746c105b71e" exitCode=0
Nov 04 00:15:45.331871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.331834    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn" event={"ID":"c123ea33-cf7c-41eb-9eca-70fd8119c4ec","Type":"ContainerDied","Data":"8d2d212610a2e06e834878103dd3a27ddba2cbfa175947bd8417b746c105b71e"}
Nov 04 00:15:45.333651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.333626    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4ee1001-32b6-46bc-9329-9a49589e28e7" containerID="00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f" exitCode=0
Nov 04 00:15:45.333795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.333659    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"
Nov 04 00:15:45.333795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.333685    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th" event={"ID":"e4ee1001-32b6-46bc-9329-9a49589e28e7","Type":"ContainerDied","Data":"00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f"}
Nov 04 00:15:45.333795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.333729    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th" event={"ID":"e4ee1001-32b6-46bc-9329-9a49589e28e7","Type":"ContainerDied","Data":"1302eafce3006ab9016c518fbd260cc1cc5bea8673a72f30c5e0b8caf3292625"}
Nov 04 00:15:45.333795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.333763    5074 scope.go:122] "RemoveContainer" containerID="00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f"
Nov 04 00:15:45.346311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.346294    5074 scope.go:122] "RemoveContainer" containerID="00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f"
Nov 04 00:15:45.346568 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:45.346547    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f\": container with ID starting with 00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f not found: ID does not exist" containerID="00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f"
Nov 04 00:15:45.346636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.346579    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f"} err="failed to get container status \"00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f\": rpc error: code = NotFound desc = could not find container \"00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f\": container with ID starting with 00afa7a62d234cde025e1c2c35aa2d93d7d95f184f8104481f722aa2a680941f not found: ID does not exist"
Nov 04 00:15:45.347434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.347414    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4fpk\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-kube-api-access-j4fpk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:45.347486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.347437    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:45.347486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.347448    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4ee1001-32b6-46bc-9329-9a49589e28e7-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:45.347486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.347458    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4ee1001-32b6-46bc-9329-9a49589e28e7-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:45.374396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.374367    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"]
Nov 04 00:15:45.380320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:45.380291    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-868c5t8th"]
Nov 04 00:15:46.544887 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.544859    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:46.564240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.564206    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca\") pod \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") "
Nov 04 00:15:46.564379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.564275    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn\" (UniqueName: \"kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn\") pod \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") "
Nov 04 00:15:46.564379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.564315    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca\") pod \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\" (UID: \"c123ea33-cf7c-41eb-9eca-70fd8119c4ec\") "
Nov 04 00:15:46.565339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.565316    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca" pod "c123ea33-cf7c-41eb-9eca-70fd8119c4ec" (UID: "c123ea33-cf7c-41eb-9eca-70fd8119c4ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:46.568309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.567590    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn" pod "c123ea33-cf7c-41eb-9eca-70fd8119c4ec" (UID: "c123ea33-cf7c-41eb-9eca-70fd8119c4ec"). InnerVolumeSpecName "kube-api-access-drncn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:46.568601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.568573    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca" pod "c123ea33-cf7c-41eb-9eca-70fd8119c4ec" (UID: "c123ea33-cf7c-41eb-9eca-70fd8119c4ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:15:46.666636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.666550    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:46.666636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.666581    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drncn\" (UniqueName: \"kubernetes.io/projected/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-kube-api-access-drncn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:46.666636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:46.666598    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c123ea33-cf7c-41eb-9eca-70fd8119c4ec-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:47.021730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:47.021695    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ee1001-32b6-46bc-9329-9a49589e28e7" path="/var/lib/kubelet/pods/e4ee1001-32b6-46bc-9329-9a49589e28e7/volumes"
Nov 04 00:15:47.356878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:47.356798    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"
Nov 04 00:15:47.359413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:47.356803    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn" event={"ID":"c123ea33-cf7c-41eb-9eca-70fd8119c4ec","Type":"ContainerDied","Data":"998d0286442f00fabffec2639bec64efebd35a7c6f8058b3bd8d45997e7f2968"}
Nov 04 00:15:47.359413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:47.356939    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="998d0286442f00fabffec2639bec64efebd35a7c6f8058b3bd8d45997e7f2968"
Nov 04 00:15:51.564801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:51.562332    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"]
Nov 04 00:15:54.980502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:54.980465    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"]
Nov 04 00:15:55.444236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:55.444200    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"]
Nov 04 00:15:56.562879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.562835    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h" podUID="0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" containerName="payload-processing" containerID="cri-o://fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288" gracePeriod=25
Nov 04 00:15:56.886125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.886095    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:15:56.966713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.966683    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume\") pod \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") "
Nov 04 00:15:56.966922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.966840    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume\") pod \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") "
Nov 04 00:15:56.966922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.966884    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials\") pod \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") "
Nov 04 00:15:56.967075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.967055    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm\") pod \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\" (UID: \"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf\") "
Nov 04 00:15:56.967217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.967192    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume" pod "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" (UID: "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:56.967348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.967327    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume" pod "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" (UID: "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:56.967696 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.967672    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:56.967858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.967699    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:56.969492 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.969468    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm" pod "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" (UID: "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf"). InnerVolumeSpecName "kube-api-access-kxbbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:56.969588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:56.969511    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials" pod "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" (UID: "0a4fe9b8-66aa-4ff8-91b7-908b164c79bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:57.069698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.069667    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:57.069698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.069696    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbbm\" (UniqueName: \"kubernetes.io/projected/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf-kube-api-access-kxbbm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:57.472057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.472021    5074 generic.go:356] "Generic (PLEG): container finished" podID="0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" containerID="fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288" exitCode=0
Nov 04 00:15:57.472280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.472079    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"
Nov 04 00:15:57.472280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.472108    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h" event={"ID":"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf","Type":"ContainerDied","Data":"fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288"}
Nov 04 00:15:57.472280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.472163    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h" event={"ID":"0a4fe9b8-66aa-4ff8-91b7-908b164c79bf","Type":"ContainerDied","Data":"e44bb52db6f83cb2982ca136ea7af5addca6eee6b319f280a1f85b63e4a74250"}
Nov 04 00:15:57.472280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.472184    5074 scope.go:122] "RemoveContainer" containerID="fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288"
Nov 04 00:15:57.484708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.484677    5074 scope.go:122] "RemoveContainer" containerID="fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288"
Nov 04 00:15:57.485056 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:57.485034    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288\": container with ID starting with fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288 not found: ID does not exist" containerID="fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288"
Nov 04 00:15:57.485126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.485063    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288"} err="failed to get container status \"fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288\": rpc error: code = NotFound desc = could not find container \"fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288\": container with ID starting with fb814f34be9562f76042fb63e0e59d74339f4ea5d2a2637a2254e373ba61b288 not found: ID does not exist"
Nov 04 00:15:57.495650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.495615    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"]
Nov 04 00:15:57.500400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:57.500378    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-587c7qc8h"]
Nov 04 00:15:58.025011 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:15:58.024973    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:15:59.073758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:59.050048    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" path="/var/lib/kubelet/pods/0a4fe9b8-66aa-4ff8-91b7-908b164c79bf/volumes"
Nov 04 00:15:59.634170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:59.634131    5074 scope.go:122] "RemoveContainer" containerID="301b95151b543f6424a52ec99d70f1975d1f5c72f9c95ba280aefac929685a2e"
Nov 04 00:15:59.666200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:59.665958    5074 scope.go:122] "RemoveContainer" containerID="d589a17d5c732f731ac60592b43bbfae7ba8e47a844f3dbc1a406fb56a92f914"
Nov 04 00:15:59.981211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:15:59.981170    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd" podUID="f5be7f02-9455-4446-b8b7-1f82ca97249f" containerName="payload-processing" containerID="cri-o://58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166" gracePeriod=25
Nov 04 00:16:00.029809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.029773    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"]
Nov 04 00:16:00.038518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.038485    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b46d674-7lmqv"]
Nov 04 00:16:00.295749 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.295712    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:16:00.324735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.324704    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume\") pod \"f5be7f02-9455-4446-b8b7-1f82ca97249f\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") "
Nov 04 00:16:00.324950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.324920    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume\") pod \"f5be7f02-9455-4446-b8b7-1f82ca97249f\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") "
Nov 04 00:16:00.325030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.324957    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z\") pod \"f5be7f02-9455-4446-b8b7-1f82ca97249f\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") "
Nov 04 00:16:00.325088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.325050    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials\") pod \"f5be7f02-9455-4446-b8b7-1f82ca97249f\" (UID: \"f5be7f02-9455-4446-b8b7-1f82ca97249f\") "
Nov 04 00:16:00.326361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.326103    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume" pod "f5be7f02-9455-4446-b8b7-1f82ca97249f" (UID: "f5be7f02-9455-4446-b8b7-1f82ca97249f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:00.326929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.326681    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume" pod "f5be7f02-9455-4446-b8b7-1f82ca97249f" (UID: "f5be7f02-9455-4446-b8b7-1f82ca97249f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:00.337234 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.337210    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z" pod "f5be7f02-9455-4446-b8b7-1f82ca97249f" (UID: "f5be7f02-9455-4446-b8b7-1f82ca97249f"). InnerVolumeSpecName "kube-api-access-thf6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:00.337357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.337331    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials" pod "f5be7f02-9455-4446-b8b7-1f82ca97249f" (UID: "f5be7f02-9455-4446-b8b7-1f82ca97249f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:00.426512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.426475    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.426512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.426507    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5be7f02-9455-4446-b8b7-1f82ca97249f-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.426512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.426518    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thf6z\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-kube-api-access-thf6z\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.426770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.426527    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5be7f02-9455-4446-b8b7-1f82ca97249f-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.445032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.444991    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv" podUID="81887362-c237-476a-a338-8c325256dc90" containerName="payload-processing" containerID="cri-o://5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13" gracePeriod=25
Nov 04 00:16:00.502257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.502220    5074 generic.go:356] "Generic (PLEG): container finished" podID="f5be7f02-9455-4446-b8b7-1f82ca97249f" containerID="58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166" exitCode=0
Nov 04 00:16:00.502427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.502275    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"
Nov 04 00:16:00.502427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.502301    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd" event={"ID":"f5be7f02-9455-4446-b8b7-1f82ca97249f","Type":"ContainerDied","Data":"58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166"}
Nov 04 00:16:00.502427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.502390    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd" event={"ID":"f5be7f02-9455-4446-b8b7-1f82ca97249f","Type":"ContainerDied","Data":"448bb0ce53649fdaad9bec58c0aefe704ffa37f2fc0c526395b046b43cfdb5fd"}
Nov 04 00:16:00.502601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.502431    5074 scope.go:122] "RemoveContainer" containerID="58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166"
Nov 04 00:16:00.541868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.541822    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"]
Nov 04 00:16:00.546291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.546257    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fa0d0a/payload-processing-external-model-e2e-extmod-a-fa0d0a-5fbb9zgjd"]
Nov 04 00:16:00.560481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.560455    5074 scope.go:122] "RemoveContainer" containerID="58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166"
Nov 04 00:16:00.560858 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:00.560823    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166\": container with ID starting with 58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166 not found: ID does not exist" containerID="58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166"
Nov 04 00:16:00.560945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.560868    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166"} err="failed to get container status \"58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166\": rpc error: code = NotFound desc = could not find container \"58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166\": container with ID starting with 58eae6ed52d6412bb7e54574bcb07a0e608c0f687f8e67ba498e398864da5166 not found: ID does not exist"
Nov 04 00:16:00.707544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.707512    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:16:00.733148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.732443    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume\") pod \"81887362-c237-476a-a338-8c325256dc90\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") "
Nov 04 00:16:00.733148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.732532    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp\") pod \"81887362-c237-476a-a338-8c325256dc90\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") "
Nov 04 00:16:00.733148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.732572    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials\") pod \"81887362-c237-476a-a338-8c325256dc90\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") "
Nov 04 00:16:00.733148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.732600    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume\") pod \"81887362-c237-476a-a338-8c325256dc90\" (UID: \"81887362-c237-476a-a338-8c325256dc90\") "
Nov 04 00:16:00.733148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.732919    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume" pod "81887362-c237-476a-a338-8c325256dc90" (UID: "81887362-c237-476a-a338-8c325256dc90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:00.733569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.733462    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.733832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.733794    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume" pod "81887362-c237-476a-a338-8c325256dc90" (UID: "81887362-c237-476a-a338-8c325256dc90"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:00.736076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.736049    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp" pod "81887362-c237-476a-a338-8c325256dc90" (UID: "81887362-c237-476a-a338-8c325256dc90"). InnerVolumeSpecName "kube-api-access-cclbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:00.736690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.736664    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials" pod "81887362-c237-476a-a338-8c325256dc90" (UID: "81887362-c237-476a-a338-8c325256dc90"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:00.834794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.834687    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclbp\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-kube-api-access-cclbp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.834794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.834719    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81887362-c237-476a-a338-8c325256dc90-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:00.834794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:00.834732    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81887362-c237-476a-a338-8c325256dc90-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:01.021479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.021448    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="649996dd-72e7-4914-9d22-b8c761f906b2" path="/var/lib/kubelet/pods/649996dd-72e7-4914-9d22-b8c761f906b2/volumes"
Nov 04 00:16:01.021999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.021984    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5be7f02-9455-4446-b8b7-1f82ca97249f" path="/var/lib/kubelet/pods/f5be7f02-9455-4446-b8b7-1f82ca97249f/volumes"
Nov 04 00:16:01.517631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.517591    5074 generic.go:356] "Generic (PLEG): container finished" podID="81887362-c237-476a-a338-8c325256dc90" containerID="5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13" exitCode=0
Nov 04 00:16:01.518141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.517656    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"
Nov 04 00:16:01.518141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.517680    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv" event={"ID":"81887362-c237-476a-a338-8c325256dc90","Type":"ContainerDied","Data":"5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13"}
Nov 04 00:16:01.518141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.517758    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv" event={"ID":"81887362-c237-476a-a338-8c325256dc90","Type":"ContainerDied","Data":"702b80097dd474ad89236394730863400305760359f4b4d19901aedacbb0119b"}
Nov 04 00:16:01.518141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.517796    5074 scope.go:122] "RemoveContainer" containerID="5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13"
Nov 04 00:16:01.532797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.532730    5074 scope.go:122] "RemoveContainer" containerID="5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13"
Nov 04 00:16:01.533152 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:01.533118    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13\": container with ID starting with 5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13 not found: ID does not exist" containerID="5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13"
Nov 04 00:16:01.533258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.533150    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13"} err="failed to get container status \"5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13\": rpc error: code = NotFound desc = could not find container \"5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13\": container with ID starting with 5b40e7ed3974fd05e241c114a751b21f924a2f7ebbac8d236f2e40741714aa13 not found: ID does not exist"
Nov 04 00:16:01.548045 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.548012    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"]
Nov 04 00:16:01.552244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:01.552220    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-76d7pq5xv"]
Nov 04 00:16:03.054761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:03.054427    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81887362-c237-476a-a338-8c325256dc90" path="/var/lib/kubelet/pods/81887362-c237-476a-a338-8c325256dc90/volumes"
Nov 04 00:16:07.940382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.940304    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"]
Nov 04 00:16:07.941204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941184    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4ee1001-32b6-46bc-9329-9a49589e28e7" containerName="payload-processing"
Nov 04 00:16:07.941259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941204    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ee1001-32b6-46bc-9329-9a49589e28e7" containerName="payload-processing"
Nov 04 00:16:07.941259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941222    5074 cpu_manager.go:414] "containerMap: removing container" podUID="81887362-c237-476a-a338-8c325256dc90" containerName="payload-processing"
Nov 04 00:16:07.941259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941227    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81887362-c237-476a-a338-8c325256dc90" containerName="payload-processing"
Nov 04 00:16:07.941259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941240    5074 cpu_manager.go:414] "containerMap: removing container" podUID="282d83c3-dd16-4fee-aefe-63e45f2f51f5" containerName="revoke-keys"
Nov 04 00:16:07.941259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941245    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282d83c3-dd16-4fee-aefe-63e45f2f51f5" containerName="revoke-keys"
Nov 04 00:16:07.941259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941259    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" containerName="payload-processing"
Nov 04 00:16:07.941462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941264    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" containerName="payload-processing"
Nov 04 00:16:07.941462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941302    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c123ea33-cf7c-41eb-9eca-70fd8119c4ec" containerName="revoke-keys"
Nov 04 00:16:07.941462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941308    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c123ea33-cf7c-41eb-9eca-70fd8119c4ec" containerName="revoke-keys"
Nov 04 00:16:07.941462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941326    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f5be7f02-9455-4446-b8b7-1f82ca97249f" containerName="payload-processing"
Nov 04 00:16:07.941462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941331    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5be7f02-9455-4446-b8b7-1f82ca97249f" containerName="payload-processing"
Nov 04 00:16:07.941462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941459    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c123ea33-cf7c-41eb-9eca-70fd8119c4ec" containerName="revoke-keys"
Nov 04 00:16:07.941653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941472    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5be7f02-9455-4446-b8b7-1f82ca97249f" containerName="payload-processing"
Nov 04 00:16:07.941653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941478    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ee1001-32b6-46bc-9329-9a49589e28e7" containerName="payload-processing"
Nov 04 00:16:07.941653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941484    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="81887362-c237-476a-a338-8c325256dc90" containerName="payload-processing"
Nov 04 00:16:07.941653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941490    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="282d83c3-dd16-4fee-aefe-63e45f2f51f5" containerName="revoke-keys"
Nov 04 00:16:07.941653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.941498    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4fe9b8-66aa-4ff8-91b7-908b164c79bf" containerName="payload-processing"
Nov 04 00:16:07.947360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.947336    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:07.990013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.989976    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"agc-cred-1d448de1557f\""
Nov 04 00:16:07.993922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:07.993889    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"]
Nov 04 00:16:08.096155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.096121    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5cj\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.096332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.096208    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.096479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.096455    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.096524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.096514    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.197813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.197709    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5cj\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.198001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.197836    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.198001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.197959    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.198133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.198001    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.198781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.198723    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.198999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.198890    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.201117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.201094    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.210916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.210888    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5cj\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.253779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.253750    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"]
Nov 04 00:16:08.259118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.259098    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.267144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.266608    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:08.267144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.266943    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"]
Nov 04 00:16:08.267372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.267154    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-4ef7391bd29d\""
Nov 04 00:16:08.401648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.401612    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.401912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.401891    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.401965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.401936    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6mg\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.402149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.402124    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.503388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.503291    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.503549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.503391    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.503619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.503601    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.503671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.503639    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6mg\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.504164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.504129    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.504318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.504220    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.506385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.506364    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.518931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.518905    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6mg\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.579704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.579657    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"]
Nov 04 00:16:08.580056 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:08.580031    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a070641_54bd_469c_87f2_43b98182c7fe.slice/crio-2b501c0b6cb672d9ed9a3ab62bdf684d62bcc4ed6989cb15606141ff7809f5b8 WatchSource:0}: Error finding container 2b501c0b6cb672d9ed9a3ab62bdf684d62bcc4ed6989cb15606141ff7809f5b8: Status 404 returned error can't find the container with id 2b501c0b6cb672d9ed9a3ab62bdf684d62bcc4ed6989cb15606141ff7809f5b8
Nov 04 00:16:08.595140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.595119    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:08.629559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.629527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m" event={"ID":"9a070641-54bd-469c-87f2-43b98182c7fe","Type":"ContainerStarted","Data":"2b501c0b6cb672d9ed9a3ab62bdf684d62bcc4ed6989cb15606141ff7809f5b8"}
Nov 04 00:16:08.898763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.898232    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"]
Nov 04 00:16:08.901585 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:08.901542    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fe9b48_b66d_46ce_8f3e_33472837a9a6.slice/crio-8c82765bbfdde416c3c29f05e256c4a019fda6991277590771e2f78500e9ea47 WatchSource:0}: Error finding container 8c82765bbfdde416c3c29f05e256c4a019fda6991277590771e2f78500e9ea47: Status 404 returned error can't find the container with id 8c82765bbfdde416c3c29f05e256c4a019fda6991277590771e2f78500e9ea47
Nov 04 00:16:08.975784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.975755    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"]
Nov 04 00:16:08.985329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.985304    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"]
Nov 04 00:16:08.985491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.985439    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:08.992663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.992635    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8cc1c0\""
Nov 04 00:16:08.998096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.996987    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"]
Nov 04 00:16:08.998096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:08.997103    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.013983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.012590    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8cc1c0-dockercfg-9f8cv\""
Nov 04 00:16:09.063928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.063595    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"]
Nov 04 00:16:09.118936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.118886    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.119448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.119389    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjx6m\" (UniqueName: \"kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m\") pod \"payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.119611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.119545    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.119611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.119588    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94br\" (UniqueName: \"kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br\") pod \"payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.226133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.226093    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.228979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.228949    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjx6m\" (UniqueName: \"kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m\") pod \"payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.229281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.229263    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.229404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.229391    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94br\" (UniqueName: \"kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br\") pod \"payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.229864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.228700    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.231018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.230995    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.252923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.252882    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjx6m\" (UniqueName: \"kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m\") pod \"payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.279820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.279784    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94br\" (UniqueName: \"kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br\") pod \"payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.349861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.349825    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:09.366138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.366111    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:09.641035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.640968    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4" event={"ID":"b4fe9b48-b66d-46ce-8f3e-33472837a9a6","Type":"ContainerStarted","Data":"3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3"}
Nov 04 00:16:09.641242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.641058    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:09.641242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.641083    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4" event={"ID":"b4fe9b48-b66d-46ce-8f3e-33472837a9a6","Type":"ContainerStarted","Data":"8c82765bbfdde416c3c29f05e256c4a019fda6991277590771e2f78500e9ea47"}
Nov 04 00:16:09.643155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.643129    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m" event={"ID":"9a070641-54bd-469c-87f2-43b98182c7fe","Type":"ContainerStarted","Data":"205c11cedc983231b2b0701402fe0d771fecdf8d46765337d9aad0788562ab1c"}
Nov 04 00:16:09.643284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.643235    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:09.653979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.653951    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"]
Nov 04 00:16:09.654261 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:09.654211    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be8c401_cee0_4195_b71e_c76b29f03954.slice/crio-82e218e958027710b52598b6d833ee2afa50197f0b843d693d4c5ecccb60b007 WatchSource:0}: Error finding container 82e218e958027710b52598b6d833ee2afa50197f0b843d693d4c5ecccb60b007: Status 404 returned error can't find the container with id 82e218e958027710b52598b6d833ee2afa50197f0b843d693d4c5ecccb60b007
Nov 04 00:16:09.667104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.667058    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4" podStartSLOduration=1.6670446810000001 podStartE2EDuration="1.667044681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:09.663588944 +0000 UTC m=+2834.763189090" watchObservedRunningTime="2026-11-04 00:16:09.667044681 +0000 UTC m=+2834.766644871"
Nov 04 00:16:09.674863 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:09.674812    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1138f268_e37d_40a9_8214_9b7af5714a37.slice/crio-04d1a910ddcdaf7748922e1a6b4d88f6e1874938a24c7c75856fc544f33420d2 WatchSource:0}: Error finding container 04d1a910ddcdaf7748922e1a6b4d88f6e1874938a24c7c75856fc544f33420d2: Status 404 returned error can't find the container with id 04d1a910ddcdaf7748922e1a6b4d88f6e1874938a24c7c75856fc544f33420d2
Nov 04 00:16:09.675711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.675687    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"]
Nov 04 00:16:09.693596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:09.693526    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m" podStartSLOduration=2.693507745 podStartE2EDuration="2.693507745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:09.690092611 +0000 UTC m=+2834.789692756" watchObservedRunningTime="2026-11-04 00:16:09.693507745 +0000 UTC m=+2834.793107890"
Nov 04 00:16:10.657615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.657574    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" event={"ID":"1be8c401-cee0-4195-b71e-c76b29f03954","Type":"ContainerStarted","Data":"46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce"}
Nov 04 00:16:10.658316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.658280    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" event={"ID":"1be8c401-cee0-4195-b71e-c76b29f03954","Type":"ContainerStarted","Data":"82e218e958027710b52598b6d833ee2afa50197f0b843d693d4c5ecccb60b007"}
Nov 04 00:16:10.658462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.658334    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:10.661623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.661558    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" event={"ID":"1138f268-e37d-40a9-8214-9b7af5714a37","Type":"ContainerStarted","Data":"b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e"}
Nov 04 00:16:10.661623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.661616    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:10.661807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.661643    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" event={"ID":"1138f268-e37d-40a9-8214-9b7af5714a37","Type":"ContainerStarted","Data":"04d1a910ddcdaf7748922e1a6b4d88f6e1874938a24c7c75856fc544f33420d2"}
Nov 04 00:16:10.692873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.689071    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" podStartSLOduration=2.689052814 podStartE2EDuration="2.689052814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:10.684485641 +0000 UTC m=+2835.784085786" watchObservedRunningTime="2026-11-04 00:16:10.689052814 +0000 UTC m=+2835.788652959"
Nov 04 00:16:10.732694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:10.732618    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" podStartSLOduration=2.73259704 podStartE2EDuration="2.73259704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:10.711229569 +0000 UTC m=+2835.810829714" watchObservedRunningTime="2026-11-04 00:16:10.73259704 +0000 UTC m=+2835.832197185"
Nov 04 00:16:11.609792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.609753    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"]
Nov 04 00:16:11.619759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.617026    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.623761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.623298    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-71b2f3\""
Nov 04 00:16:11.623761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.623397    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"routing-overlay\""
Nov 04 00:16:11.623761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.623466    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"openshift-service-ca.crt\""
Nov 04 00:16:11.625155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.625119    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"agc-cred-fff38664de1c\""
Nov 04 00:16:11.625280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.625208    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-71b2f3\"/\"kube-root-ca.crt\""
Nov 04 00:16:11.626709 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.626685    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"]
Nov 04 00:16:11.694248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.694210    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsnt\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.694753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.694288    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.694753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.694448    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.694753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.694590    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.813818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.796419    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.813818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.796596    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsnt\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.813818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.796665    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.813818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.797183    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.813818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.799070    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.813818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.800327    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.861219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.848884    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.874761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.865328    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsnt\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt\") pod \"payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") " pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:11.946059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:11.946018    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:12.128458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.128377    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"]
Nov 04 00:16:12.138040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.137997    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.149252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.149184    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"]
Nov 04 00:16:12.149883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.149668    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"routing-overlay\""
Nov 04 00:16:12.149883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.149811    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"payload-processing-external-model-e2e--36d07d4b-dockercfg-7nfvs\""
Nov 04 00:16:12.149883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.149853    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"agc-cred-b35f0d071f6b\""
Nov 04 00:16:12.150125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.149972    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"openshift-service-ca.crt\""
Nov 04 00:16:12.150125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.150094    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"kube-root-ca.crt\""
Nov 04 00:16:12.150245 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.150175    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7d218\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c7d218\""
Nov 04 00:16:12.221877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.221790    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.221877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.221849    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.222174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.221942    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmw6\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.222174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.222015    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.282929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.282768    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"]
Nov 04 00:16:12.285081 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:12.284095    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbec92c4_812e_4e61_a05a_d13fa8946fc0.slice/crio-e3355fe793ec3ce2b9ca0affd9debab58eb66fff295567a310cf1981ce503975 WatchSource:0}: Error finding container e3355fe793ec3ce2b9ca0affd9debab58eb66fff295567a310cf1981ce503975: Status 404 returned error can't find the container with id e3355fe793ec3ce2b9ca0affd9debab58eb66fff295567a310cf1981ce503975
Nov 04 00:16:12.328439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.328396    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.328643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.328623    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.328692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.328679    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.328811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.328793    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmw6\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.329419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.329392    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.329862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.329834    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.331988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.331963    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.342548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.342527    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmw6\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6\") pod \"payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") " pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.482541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.482501    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:12.695891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.695839    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" event={"ID":"cbec92c4-812e-4e61-a05a-d13fa8946fc0","Type":"ContainerStarted","Data":"5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc"}
Nov 04 00:16:12.696433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.695944    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" event={"ID":"cbec92c4-812e-4e61-a05a-d13fa8946fc0","Type":"ContainerStarted","Data":"e3355fe793ec3ce2b9ca0affd9debab58eb66fff295567a310cf1981ce503975"}
Nov 04 00:16:12.696433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.696285    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:12.721323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.721277    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" podStartSLOduration=1.721264178 podStartE2EDuration="1.721264178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:12.718003765 +0000 UTC m=+2837.817603910" watchObservedRunningTime="2026-11-04 00:16:12.721264178 +0000 UTC m=+2837.820864352"
Nov 04 00:16:12.777398 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:12.777368    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759552ae_8014_45f3_a8a1_5c6875d004d1.slice/crio-f7fa196cb06fb6cc648cea2890f0150568fc9909be5f12352385ccaf55363205 WatchSource:0}: Error finding container f7fa196cb06fb6cc648cea2890f0150568fc9909be5f12352385ccaf55363205: Status 404 returned error can't find the container with id f7fa196cb06fb6cc648cea2890f0150568fc9909be5f12352385ccaf55363205
Nov 04 00:16:12.777818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:12.777793    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"]
Nov 04 00:16:13.403290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:13.398523    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"]
Nov 04 00:16:13.744497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:13.742836    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" event={"ID":"759552ae-8014-45f3-a8a1-5c6875d004d1","Type":"ContainerStarted","Data":"2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba"}
Nov 04 00:16:13.744497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:13.742918    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" event={"ID":"759552ae-8014-45f3-a8a1-5c6875d004d1","Type":"ContainerStarted","Data":"f7fa196cb06fb6cc648cea2890f0150568fc9909be5f12352385ccaf55363205"}
Nov 04 00:16:13.744497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:13.743314    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:13.766786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:13.766728    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" podStartSLOduration=1.7667167940000001 podStartE2EDuration="1.766716794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:13.765550222 +0000 UTC m=+2838.865150366" watchObservedRunningTime="2026-11-04 00:16:13.766716794 +0000 UTC m=+2838.866316965"
Nov 04 00:16:13.853414 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.853378    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:13.853613 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.853444    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:14.353429055 +0000 UTC m=+2839.453029178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:13.853943 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.853918    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-b35f0d071f6b: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:13.853943 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.853943    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:13.854149 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.853964    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-fff38664de1c: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:13.854149 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.853988    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:13.854149 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.854000    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:14.353981938 +0000 UTC m=+2839.453582061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:13.854149 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:13.854044    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:14.354026873 +0000 UTC m=+2839.453627007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:13.893325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:13.893287    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"]
Nov 04 00:16:14.364781 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364718    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-b35f0d071f6b: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:14.364781 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364776    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:14.365004 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364774    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:14.365004 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364841    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-fff38664de1c: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:14.365004 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364855    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:14.365004 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364824    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:15.364809838 +0000 UTC m=+2840.464409960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:14.365004 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364888    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:15.364872854 +0000 UTC m=+2840.464472989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:14.365004 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.364905    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:15.36489644 +0000 UTC m=+2840.464496575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:14.754585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:14.754552    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" secret="" err="secret \"payload-processing-external-model-e2e--36d07d4b-dockercfg-7nfvs\" not found"
Nov 04 00:16:14.872800 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.872761    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-plugins-e2e-extmod-b-c7d218: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:14.872989 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:14.872854    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:15.372834473 +0000 UTC m=+2840.472434612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:15.028409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:15.028320    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"]
Nov 04 00:16:15.033014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:15.032981    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7d32a3b-qbwng"]
Nov 04 00:16:15.380776 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380655    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-plugins-e2e-extmod-b-c7d218: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:15.380776 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380732    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:16.380717984 +0000 UTC m=+2841.480318135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:15.380776 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380760    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-fff38664de1c: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380786    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380808    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-b35f0d071f6b: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380827    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380832    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:17.380820206 +0000 UTC m=+2842.480420350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380843    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380866    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:17.38085364 +0000 UTC m=+2842.480453782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:15.381064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:15.380883    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:17.380874403 +0000 UTC m=+2842.480474528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:16.401847 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:16.401793    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-plugins-e2e-extmod-b-c7d218: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:16.402275 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:16.401902    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:18.401882294 +0000 UTC m=+2843.501482425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:17.025480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:17.025435    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9314252-787b-4a9b-9b1b-7306f298f8bb" path="/var/lib/kubelet/pods/d9314252-787b-4a9b-9b1b-7306f298f8bb/volumes"
Nov 04 00:16:17.418481 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418388    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-plugins-e2e-extmod-a-71b2f3: configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:17.418481 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418458    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:21.418444358 +0000 UTC m=+2846.518044480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-71b2f3" not found
Nov 04 00:16:17.419094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418525    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-71b2f3/agc-cred-fff38664de1c: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:17.419094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418545    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w: secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:17.419094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418584    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials podName:cbec92c4-812e-4e61-a05a-d13fa8946fc0 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:21.4185738 +0000 UTC m=+2846.518173922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0") : secret "agc-cred-fff38664de1c" not found
Nov 04 00:16:17.419094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418712    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7d218/agc-cred-b35f0d071f6b: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:17.419094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418732    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn: secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:17.419094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:17.418798    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:21.418783101 +0000 UTC m=+2846.518383226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : secret "agc-cred-b35f0d071f6b" not found
Nov 04 00:16:18.434556 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:18.434517    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-plugins-e2e-extmod-b-c7d218: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:18.435094 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:18.434605    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume podName:759552ae-8014-45f3-a8a1-5c6875d004d1 nodeName:}" failed. No retries permitted until 2026-11-04 00:16:22.434586453 +0000 UTC m=+2847.534186599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7d218" not found
Nov 04 00:16:19.754302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:19.754261    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" podUID="cbec92c4-812e-4e61-a05a-d13fa8946fc0" containerName="payload-processing" containerID="cri-o://5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc" gracePeriod=25
Nov 04 00:16:20.052477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.052441    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:20.163107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.163066    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume\") pod \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") "
Nov 04 00:16:20.163323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.163194    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt\") pod \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") "
Nov 04 00:16:20.163323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.163275    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume\") pod \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") "
Nov 04 00:16:20.163472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.163447    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials\") pod \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\" (UID: \"cbec92c4-812e-4e61-a05a-d13fa8946fc0\") "
Nov 04 00:16:20.163605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.163579    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume" pod "cbec92c4-812e-4e61-a05a-d13fa8946fc0" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:20.163688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.163664    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume" pod "cbec92c4-812e-4e61-a05a-d13fa8946fc0" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:20.164446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.164419    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:20.164446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.164448    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec92c4-812e-4e61-a05a-d13fa8946fc0-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:20.165876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.165843    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt" pod "cbec92c4-812e-4e61-a05a-d13fa8946fc0" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0"). InnerVolumeSpecName "kube-api-access-sgsnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:20.165982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.165882    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials" pod "cbec92c4-812e-4e61-a05a-d13fa8946fc0" (UID: "cbec92c4-812e-4e61-a05a-d13fa8946fc0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:20.265919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.265834    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsnt\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-kube-api-access-sgsnt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:20.265919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.265868    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbec92c4-812e-4e61-a05a-d13fa8946fc0-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:20.665199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.665116    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:16:20.665199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.665186    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:16:20.734792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.732586    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"]
Nov 04 00:16:20.746911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.746875    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"]
Nov 04 00:16:20.763466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.763428    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" podUID="759552ae-8014-45f3-a8a1-5c6875d004d1" containerName="payload-processing" containerID="cri-o://2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba" gracePeriod=25
Nov 04 00:16:20.821196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.821156    5074 generic.go:356] "Generic (PLEG): container finished" podID="cbec92c4-812e-4e61-a05a-d13fa8946fc0" containerID="5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc" exitCode=0
Nov 04 00:16:20.821343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.821239    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"
Nov 04 00:16:20.821343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.821240    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" event={"ID":"cbec92c4-812e-4e61-a05a-d13fa8946fc0","Type":"ContainerDied","Data":"5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc"}
Nov 04 00:16:20.821455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.821433    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w" event={"ID":"cbec92c4-812e-4e61-a05a-d13fa8946fc0","Type":"ContainerDied","Data":"e3355fe793ec3ce2b9ca0affd9debab58eb66fff295567a310cf1981ce503975"}
Nov 04 00:16:20.821503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.821464    5074 scope.go:122] "RemoveContainer" containerID="5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc"
Nov 04 00:16:20.880255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.880152    5074 scope.go:122] "RemoveContainer" containerID="5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc"
Nov 04 00:16:20.880544 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:20.880522    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc\": container with ID starting with 5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc not found: ID does not exist" containerID="5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc"
Nov 04 00:16:20.880637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.880556    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc"} err="failed to get container status \"5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc\": rpc error: code = NotFound desc = could not find container \"5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc\": container with ID starting with 5e15018513cbb9654897ca01278b6cf226810f3359a45df9eb0e26a4604ef7dc not found: ID does not exist"
Nov 04 00:16:20.886851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.886812    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"]
Nov 04 00:16:20.895842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:20.895818    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-71b2f3/payload-processing-external-model-e2e-extmod-a-71b2f3-7fcblx99w"]
Nov 04 00:16:21.022785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.022701    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbec92c4-812e-4e61-a05a-d13fa8946fc0" path="/var/lib/kubelet/pods/cbec92c4-812e-4e61-a05a-d13fa8946fc0/volumes"
Nov 04 00:16:21.070468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.070441    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:21.180861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.180816    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume\") pod \"759552ae-8014-45f3-a8a1-5c6875d004d1\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") "
Nov 04 00:16:21.181044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.180882    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume\") pod \"759552ae-8014-45f3-a8a1-5c6875d004d1\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") "
Nov 04 00:16:21.181044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.180946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6\") pod \"759552ae-8014-45f3-a8a1-5c6875d004d1\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") "
Nov 04 00:16:21.181145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.181078    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials\") pod \"759552ae-8014-45f3-a8a1-5c6875d004d1\" (UID: \"759552ae-8014-45f3-a8a1-5c6875d004d1\") "
Nov 04 00:16:21.181327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.181283    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume" pod "759552ae-8014-45f3-a8a1-5c6875d004d1" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:21.181327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.181317    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume" pod "759552ae-8014-45f3-a8a1-5c6875d004d1" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:21.181807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.181731    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:21.181807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.181777    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/759552ae-8014-45f3-a8a1-5c6875d004d1-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:21.183320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.183294    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials" pod "759552ae-8014-45f3-a8a1-5c6875d004d1" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:21.183481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.183460    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6" pod "759552ae-8014-45f3-a8a1-5c6875d004d1" (UID: "759552ae-8014-45f3-a8a1-5c6875d004d1"). InnerVolumeSpecName "kube-api-access-2dmw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:21.284814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.283865    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dmw6\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-kube-api-access-2dmw6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:21.284814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.283902    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/759552ae-8014-45f3-a8a1-5c6875d004d1-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:21.690380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.682889    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:16:21.872117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.871934    5074 generic.go:356] "Generic (PLEG): container finished" podID="759552ae-8014-45f3-a8a1-5c6875d004d1" containerID="2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba" exitCode=0
Nov 04 00:16:21.872601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.872320    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" event={"ID":"759552ae-8014-45f3-a8a1-5c6875d004d1","Type":"ContainerDied","Data":"2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba"}
Nov 04 00:16:21.872601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.872382    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn" event={"ID":"759552ae-8014-45f3-a8a1-5c6875d004d1","Type":"ContainerDied","Data":"f7fa196cb06fb6cc648cea2890f0150568fc9909be5f12352385ccaf55363205"}
Nov 04 00:16:21.872601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.872416    5074 scope.go:122] "RemoveContainer" containerID="2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba"
Nov 04 00:16:21.872601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.872542    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"
Nov 04 00:16:21.973452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.972914    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"]
Nov 04 00:16:21.979113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.978618    5074 scope.go:122] "RemoveContainer" containerID="2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba"
Nov 04 00:16:21.979113 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:21.979014    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba\": container with ID starting with 2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba not found: ID does not exist" containerID="2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba"
Nov 04 00:16:21.979113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.979048    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba"} err="failed to get container status \"2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba\": rpc error: code = NotFound desc = could not find container \"2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba\": container with ID starting with 2d1f25fb4532757f0533ee9d8e4eb446d910d77338ea981cf00a047035b8a8ba not found: ID does not exist"
Nov 04 00:16:21.981278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:21.981232    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7d218/payload-processing-external-model-e2e-extmod-b-c7d218-7777q9vvn"]
Nov 04 00:16:22.699401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:22.699364    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:16:23.025004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:23.024910    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759552ae-8014-45f3-a8a1-5c6875d004d1" path="/var/lib/kubelet/pods/759552ae-8014-45f3-a8a1-5c6875d004d1/volumes"
Nov 04 00:16:25.733769 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:25.733708    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9" podUID="332c0e88-3f29-4d3a-a2a0-e8abbd478a38" containerName="payload-processing" containerID="cri-o://e8bb90c3ce10807bcf8bae947f980ae44ca20d8625762f37ba8536277c42aad8" gracePeriod=25
Nov 04 00:16:25.747832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:25.747793    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x" podUID="97b9fe83-fb50-412d-8c8c-d525c73c2be3" containerName="payload-processing" containerID="cri-o://f0b0d04e7dfedd3e8021b83ec0b4976924ef58181f5fd6b788f04caf23608b45" gracePeriod=25
Nov 04 00:16:25.925165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:25.924636    5074 generic.go:356] "Generic (PLEG): container finished" podID="97b9fe83-fb50-412d-8c8c-d525c73c2be3" containerID="f0b0d04e7dfedd3e8021b83ec0b4976924ef58181f5fd6b788f04caf23608b45" exitCode=0
Nov 04 00:16:25.925165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:25.924720    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x" event={"ID":"97b9fe83-fb50-412d-8c8c-d525c73c2be3","Type":"ContainerDied","Data":"f0b0d04e7dfedd3e8021b83ec0b4976924ef58181f5fd6b788f04caf23608b45"}
Nov 04 00:16:25.931238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:25.926730    5074 generic.go:356] "Generic (PLEG): container finished" podID="332c0e88-3f29-4d3a-a2a0-e8abbd478a38" containerID="e8bb90c3ce10807bcf8bae947f980ae44ca20d8625762f37ba8536277c42aad8" exitCode=0
Nov 04 00:16:25.931238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:25.926885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9" event={"ID":"332c0e88-3f29-4d3a-a2a0-e8abbd478a38","Type":"ContainerDied","Data":"e8bb90c3ce10807bcf8bae947f980ae44ca20d8625762f37ba8536277c42aad8"}
Nov 04 00:16:26.109210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.108336    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:16:26.179330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.179237    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:16:26.197353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.197311    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x\") pod \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") "
Nov 04 00:16:26.197566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.197452    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume\") pod \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") "
Nov 04 00:16:26.197566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.197520    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials\") pod \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") "
Nov 04 00:16:26.197697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.197606    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume\") pod \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\" (UID: \"332c0e88-3f29-4d3a-a2a0-e8abbd478a38\") "
Nov 04 00:16:26.204782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.200396    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume" pod "332c0e88-3f29-4d3a-a2a0-e8abbd478a38" (UID: "332c0e88-3f29-4d3a-a2a0-e8abbd478a38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:26.204782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.201864    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x" pod "332c0e88-3f29-4d3a-a2a0-e8abbd478a38" (UID: "332c0e88-3f29-4d3a-a2a0-e8abbd478a38"). InnerVolumeSpecName "kube-api-access-5kq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:26.204782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.202227    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials" pod "332c0e88-3f29-4d3a-a2a0-e8abbd478a38" (UID: "332c0e88-3f29-4d3a-a2a0-e8abbd478a38"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:26.204782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.202461    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume" pod "332c0e88-3f29-4d3a-a2a0-e8abbd478a38" (UID: "332c0e88-3f29-4d3a-a2a0-e8abbd478a38"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:26.299407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.299307    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs\") pod \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") "
Nov 04 00:16:26.299407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.299387    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume\") pod \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") "
Nov 04 00:16:26.299661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.299419    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials\") pod \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") "
Nov 04 00:16:26.299661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.299472    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume\") pod \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\" (UID: \"97b9fe83-fb50-412d-8c8c-d525c73c2be3\") "
Nov 04 00:16:26.300007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.299988    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.300097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.300010    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.300097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.300023    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.300097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.300032    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/332c0e88-3f29-4d3a-a2a0-e8abbd478a38-kube-api-access-5kq7x\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.300269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.300087    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume" pod "97b9fe83-fb50-412d-8c8c-d525c73c2be3" (UID: "97b9fe83-fb50-412d-8c8c-d525c73c2be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:26.300374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.300356    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume" pod "97b9fe83-fb50-412d-8c8c-d525c73c2be3" (UID: "97b9fe83-fb50-412d-8c8c-d525c73c2be3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:16:26.302659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.302632    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs" pod "97b9fe83-fb50-412d-8c8c-d525c73c2be3" (UID: "97b9fe83-fb50-412d-8c8c-d525c73c2be3"). InnerVolumeSpecName "kube-api-access-4d2xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:26.302892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.302824    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials" pod "97b9fe83-fb50-412d-8c8c-d525c73c2be3" (UID: "97b9fe83-fb50-412d-8c8c-d525c73c2be3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:16:26.408168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.408072    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d2xs\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-kube-api-access-4d2xs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.408168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.408113    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.408168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.408126    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b9fe83-fb50-412d-8c8c-d525c73c2be3-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.408168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.408138    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b9fe83-fb50-412d-8c8c-d525c73c2be3-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:16:26.416804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.413089    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"]
Nov 04 00:16:26.419028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.418936    5074 cpu_manager.go:414] "containerMap: removing container" podUID="cbec92c4-812e-4e61-a05a-d13fa8946fc0" containerName="payload-processing"
Nov 04 00:16:26.419244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419227    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbec92c4-812e-4e61-a05a-d13fa8946fc0" containerName="payload-processing"
Nov 04 00:16:26.419416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419397    5074 cpu_manager.go:414] "containerMap: removing container" podUID="332c0e88-3f29-4d3a-a2a0-e8abbd478a38" containerName="payload-processing"
Nov 04 00:16:26.419502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419491    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332c0e88-3f29-4d3a-a2a0-e8abbd478a38" containerName="payload-processing"
Nov 04 00:16:26.419583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419563    5074 cpu_manager.go:414] "containerMap: removing container" podUID="759552ae-8014-45f3-a8a1-5c6875d004d1" containerName="payload-processing"
Nov 04 00:16:26.419660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419641    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759552ae-8014-45f3-a8a1-5c6875d004d1" containerName="payload-processing"
Nov 04 00:16:26.419778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419758    5074 cpu_manager.go:414] "containerMap: removing container" podUID="97b9fe83-fb50-412d-8c8c-d525c73c2be3" containerName="payload-processing"
Nov 04 00:16:26.419887 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.419876    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b9fe83-fb50-412d-8c8c-d525c73c2be3" containerName="payload-processing"
Nov 04 00:16:26.420263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.420249    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="332c0e88-3f29-4d3a-a2a0-e8abbd478a38" containerName="payload-processing"
Nov 04 00:16:26.420398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.420374    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="759552ae-8014-45f3-a8a1-5c6875d004d1" containerName="payload-processing"
Nov 04 00:16:26.420505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.420495    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b9fe83-fb50-412d-8c8c-d525c73c2be3" containerName="payload-processing"
Nov 04 00:16:26.420756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.420600    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbec92c4-812e-4e61-a05a-d13fa8946fc0" containerName="payload-processing"
Nov 04 00:16:26.428801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.427681    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.434530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.433693    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c7d218-openshift-default-dockercfg-z5xv6\""
Nov 04 00:16:26.447099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.447066    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"]
Nov 04 00:16:26.611411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611330    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.611411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611392    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.611645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611464    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.611645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611538    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.611780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611652    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88bqs\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.611854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611807    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.611929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.611903    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.612068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.612053    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.612111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.612089    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.713822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.713780    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bqs\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.713845    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.713901    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714035    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714075    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714145    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714195    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714224    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714389    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.714550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714440    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.715007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714958    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.715007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.714985    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.715318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.715293    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.716894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.716867    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.717020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.716985    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.728772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.727309    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.728772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.727374    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88bqs\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.781479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.778896    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:26.951480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.951438    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x" event={"ID":"97b9fe83-fb50-412d-8c8c-d525c73c2be3","Type":"ContainerDied","Data":"351248bd8df8f5672742d012769aef9c4b52bcef1f68c24b45a197b24f2c387f"}
Nov 04 00:16:26.951682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.951515    5074 scope.go:122] "RemoveContainer" containerID="f0b0d04e7dfedd3e8021b83ec0b4976924ef58181f5fd6b788f04caf23608b45"
Nov 04 00:16:26.951682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.951651    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"
Nov 04 00:16:26.962190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.961073    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9" event={"ID":"332c0e88-3f29-4d3a-a2a0-e8abbd478a38","Type":"ContainerDied","Data":"4f9c41379e3431721926f8956a45f44feff8ab4409f92599e8b62c2cd2a740dc"}
Nov 04 00:16:26.962190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.961203    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"
Nov 04 00:16:26.980809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.980782    5074 scope.go:122] "RemoveContainer" containerID="e8bb90c3ce10807bcf8bae947f980ae44ca20d8625762f37ba8536277c42aad8"
Nov 04 00:16:26.992812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.991300    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"]
Nov 04 00:16:26.998308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:26.998070    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-7457dl58x"]
Nov 04 00:16:27.021276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.021245    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b9fe83-fb50-412d-8c8c-d525c73c2be3" path="/var/lib/kubelet/pods/97b9fe83-fb50-412d-8c8c-d525c73c2be3/volumes"
Nov 04 00:16:27.021628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.021614    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"]
Nov 04 00:16:27.024927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.024904    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf45ntz9"]
Nov 04 00:16:27.124578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.124542    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"]
Nov 04 00:16:27.133764 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:27.133657    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f32ec71_4f6a_49c4_9ba5_f6c931e159d7.slice/crio-47d86b2198484faee6af7123a4fa5c90df36393652eadf80295e4014a7776eaf WatchSource:0}: Error finding container 47d86b2198484faee6af7123a4fa5c90df36393652eadf80295e4014a7776eaf: Status 404 returned error can't find the container with id 47d86b2198484faee6af7123a4fa5c90df36393652eadf80295e4014a7776eaf
Nov 04 00:16:27.139369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.138925    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:16:27.139369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.139019    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:16:27.139369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.139066    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:16:27.366730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.366694    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"]
Nov 04 00:16:27.371857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.371795    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.381143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.381118    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9a6f7a-dockercfg-f5qsb\""
Nov 04 00:16:27.381306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.381147    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9a6f7a\""
Nov 04 00:16:27.383228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.383202    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"]
Nov 04 00:16:27.402248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.402109    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"]
Nov 04 00:16:27.409794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.409755    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.426190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.424032    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"]
Nov 04 00:16:27.435678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.435644    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.435831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.435805    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzks\" (UniqueName: \"kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks\") pod \"payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.435902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.435858    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9zj\" (UniqueName: \"kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj\") pod \"payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.435902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.435882    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.570177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.554968    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9zj\" (UniqueName: \"kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj\") pod \"payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.574647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.572894    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.575397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.573840    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.576117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.575856    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.578251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.578201    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzks\" (UniqueName: \"kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks\") pod \"payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.578569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.576941    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.623382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.616563    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzks\" (UniqueName: \"kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks\") pod \"payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.623382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.617250    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9zj\" (UniqueName: \"kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj\") pod \"payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.702868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.702828    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:27.731702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.731674    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:27.984222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.984165    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" event={"ID":"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7","Type":"ContainerStarted","Data":"4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273"}
Nov 04 00:16:27.984884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:27.984854    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" event={"ID":"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7","Type":"ContainerStarted","Data":"47d86b2198484faee6af7123a4fa5c90df36393652eadf80295e4014a7776eaf"}
Nov 04 00:16:28.052865 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.048349    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podStartSLOduration=2.048325557 podStartE2EDuration="2.048325557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:28.015268624 +0000 UTC m=+2853.114868770" watchObservedRunningTime="2026-11-04 00:16:28.048325557 +0000 UTC m=+2853.147925703"
Nov 04 00:16:28.060183 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:28.060145    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:16:28.060933 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:28.060904    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77a574c_4f20_42c2_be0e_94c8ce71b384.slice/crio-ad43944c924b91ebbe106c89466dc27bdcbb8a4d267db52afbd3edcce9275731 WatchSource:0}: Error finding container ad43944c924b91ebbe106c89466dc27bdcbb8a4d267db52afbd3edcce9275731: Status 404 returned error can't find the container with id ad43944c924b91ebbe106c89466dc27bdcbb8a4d267db52afbd3edcce9275731
Nov 04 00:16:28.061041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.060950    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"]
Nov 04 00:16:28.070909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.070886    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"]
Nov 04 00:16:28.082682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.078368    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d9e99-wvb9z"]
Nov 04 00:16:28.085805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.085778    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"]
Nov 04 00:16:28.088543 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:28.088515    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc36f37_efbc_4342_b9ff_3cd5e8d630a7.slice/crio-f812cf72bde3fb9ce42aa5d0f2ca53ee76ad537154780b476f6e9fe4d54c1617 WatchSource:0}: Error finding container f812cf72bde3fb9ce42aa5d0f2ca53ee76ad537154780b476f6e9fe4d54c1617: Status 404 returned error can't find the container with id f812cf72bde3fb9ce42aa5d0f2ca53ee76ad537154780b476f6e9fe4d54c1617
Nov 04 00:16:28.779240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.779200    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:28.781099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.781072    5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Nov 04 00:16:28.781226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:28.781130    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Nov 04 00:16:29.016996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.016949    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" event={"ID":"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7","Type":"ContainerStarted","Data":"a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea"}
Nov 04 00:16:29.017603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.017577    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" event={"ID":"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7","Type":"ContainerStarted","Data":"f812cf72bde3fb9ce42aa5d0f2ca53ee76ad537154780b476f6e9fe4d54c1617"}
Nov 04 00:16:29.027816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.018827    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:29.034162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.034097    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:29.054161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.054126    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332c0e88-3f29-4d3a-a2a0-e8abbd478a38" path="/var/lib/kubelet/pods/332c0e88-3f29-4d3a-a2a0-e8abbd478a38/volumes"
Nov 04 00:16:29.054751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.054713    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9299a51c-bba7-424b-8f84-33acfd38ff31" path="/var/lib/kubelet/pods/9299a51c-bba7-424b-8f84-33acfd38ff31/volumes"
Nov 04 00:16:29.055192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.055168    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" event={"ID":"c77a574c-4f20-42c2-be0e-94c8ce71b384","Type":"ContainerStarted","Data":"e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09"}
Nov 04 00:16:29.055293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.055217    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" event={"ID":"c77a574c-4f20-42c2-be0e-94c8ce71b384","Type":"ContainerStarted","Data":"ad43944c924b91ebbe106c89466dc27bdcbb8a4d267db52afbd3edcce9275731"}
Nov 04 00:16:29.067857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.067813    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" podStartSLOduration=2.067800308 podStartE2EDuration="2.067800308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:29.045073536 +0000 UTC m=+2854.144673691" watchObservedRunningTime="2026-11-04 00:16:29.067800308 +0000 UTC m=+2854.167400453"
Nov 04 00:16:29.069236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.069190    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" podStartSLOduration=2.0691769190000002 podStartE2EDuration="2.069176919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:29.064631362 +0000 UTC m=+2854.164231507" watchObservedRunningTime="2026-11-04 00:16:29.069176919 +0000 UTC m=+2854.168777064"
Nov 04 00:16:29.779990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.779954    5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Nov 04 00:16:29.780178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:29.780012    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Nov 04 00:16:30.779856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:30.779813    5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Nov 04 00:16:30.780286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:30.779881    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Nov 04 00:16:31.779523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:31.779484    5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Nov 04 00:16:31.779697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:31.779539    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Nov 04 00:16:32.780711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:32.780250    5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Nov 04 00:16:32.780711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:32.780321    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Nov 04 00:16:33.801663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:33.801628    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:33.802294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:33.802271    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:33.803237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:33.803208    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:16:40.269918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.269882    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"]
Nov 04 00:16:40.276076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.276055    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.286768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.283941    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"]
Nov 04 00:16:40.286768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.285524    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-954cd9\""
Nov 04 00:16:40.286768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.285658    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-954cd9-dockercfg-m9t4l\""
Nov 04 00:16:40.291660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.291630    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"]
Nov 04 00:16:40.325760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.311400    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.346014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.333627    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"]
Nov 04 00:16:40.346014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.336420    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqst\" (UniqueName: \"kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst\") pod \"payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") " pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.346014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.336484    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") " pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.346014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.336637    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.346014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.336786    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78fw\" (UniqueName: \"kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw\") pod \"payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.456779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.456688    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqst\" (UniqueName: \"kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst\") pod \"payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") " pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.456779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.456769    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") " pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.457054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.456846    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.457054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.456951    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78fw\" (UniqueName: \"kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw\") pod \"payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.479582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.458087    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") " pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.479582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.458137    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.538730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.538652    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqst\" (UniqueName: \"kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst\") pod \"payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") " pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.544226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.544195    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78fw\" (UniqueName: \"kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw\") pod \"payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.635639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.635597    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:40.641912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.641883    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:40.942259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.942231    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"]
Nov 04 00:16:40.942550 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:40.942527    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a40e38_20a8_4ae3_81e6_480811a4049b.slice/crio-e70f0bb838e469853a92ad0c53f7f2f1539413d39626af9953245f559025c810 WatchSource:0}: Error finding container e70f0bb838e469853a92ad0c53f7f2f1539413d39626af9953245f559025c810: Status 404 returned error can't find the container with id e70f0bb838e469853a92ad0c53f7f2f1539413d39626af9953245f559025c810
Nov 04 00:16:40.958870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:40.958845    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"]
Nov 04 00:16:40.960342 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:16:40.960319    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3d0fed_c06e_4eab_bcb1_9840298d2404.slice/crio-20b3eba2050dcaed489a0f80191e7a6feebad52d1fc09604e4f31896c38227d6 WatchSource:0}: Error finding container 20b3eba2050dcaed489a0f80191e7a6feebad52d1fc09604e4f31896c38227d6: Status 404 returned error can't find the container with id 20b3eba2050dcaed489a0f80191e7a6feebad52d1fc09604e4f31896c38227d6
Nov 04 00:16:41.054508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.054475    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:16:41.055440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.054723    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:16:41.187882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.187783    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7" event={"ID":"9c3d0fed-c06e-4eab-bcb1-9840298d2404","Type":"ContainerStarted","Data":"4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8"}
Nov 04 00:16:41.187882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.187848    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7" event={"ID":"9c3d0fed-c06e-4eab-bcb1-9840298d2404","Type":"ContainerStarted","Data":"20b3eba2050dcaed489a0f80191e7a6feebad52d1fc09604e4f31896c38227d6"}
Nov 04 00:16:41.187882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.187875    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:41.190104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.190071    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" event={"ID":"40a40e38-20a8-4ae3-81e6-480811a4049b","Type":"ContainerStarted","Data":"4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72"}
Nov 04 00:16:41.190242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.190126    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" event={"ID":"40a40e38-20a8-4ae3-81e6-480811a4049b","Type":"ContainerStarted","Data":"e70f0bb838e469853a92ad0c53f7f2f1539413d39626af9953245f559025c810"}
Nov 04 00:16:41.190242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.190168    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:41.219474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.219422    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7" podStartSLOduration=1.219405036 podStartE2EDuration="1.219405036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:41.217379594 +0000 UTC m=+2866.316979739" watchObservedRunningTime="2026-11-04 00:16:41.219405036 +0000 UTC m=+2866.319005179"
Nov 04 00:16:41.243276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:41.243191    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" podStartSLOduration=1.243178428 podStartE2EDuration="1.243178428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:41.23802731 +0000 UTC m=+2866.337627454" watchObservedRunningTime="2026-11-04 00:16:41.243178428 +0000 UTC m=+2866.342778572"
Nov 04 00:16:45.045138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:45.045095    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"]
Nov 04 00:16:45.049764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:45.049707    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fad6aad-dbg6g"]
Nov 04 00:16:47.022051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:47.022014    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa686443-8666-4e78-a8d5-66e851c7783a" path="/var/lib/kubelet/pods/fa686443-8666-4e78-a8d5-66e851c7783a/volumes"
Nov 04 00:16:52.203143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:52.203110    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:16:52.203795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:52.203186    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:16:56.029749 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:56.029696    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"]
Nov 04 00:16:56.034592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:56.034563    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b8e35a-9xwdb"]
Nov 04 00:16:56.038400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:56.038371    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"]
Nov 04 00:16:56.041978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:56.041952    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b3eaae-hv7rs"]
Nov 04 00:16:57.026318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:57.026283    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a507c6-2a8a-4e46-b3ec-1dd392193c04" path="/var/lib/kubelet/pods/83a507c6-2a8a-4e46-b3ec-1dd392193c04/volumes"
Nov 04 00:16:57.027070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:57.027043    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c" path="/var/lib/kubelet/pods/b2fdcb9b-2b8d-4980-9b43-c16d9ce0f02c/volumes"
Nov 04 00:16:58.027771 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:16:58.027709    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:16:59.867882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:59.867844    5074 scope.go:122] "RemoveContainer" containerID="f0a5eb4bbc2727d748deea1c4dcc0758fb0286c7bcff45811587903491a56e02"
Nov 04 00:16:59.879253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:59.879231    5074 scope.go:122] "RemoveContainer" containerID="e9ffbc4b2007fb13d694f48bcc4b025d547d148a249b784041955cd6f22d4db1"
Nov 04 00:16:59.889771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:59.889722    5074 scope.go:122] "RemoveContainer" containerID="fc6b404483bc6e43d75ca3684335d650eb310d23a75c6cc59de243e82fdfc2d5"
Nov 04 00:16:59.899633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:59.899608    5074 scope.go:122] "RemoveContainer" containerID="a7fc765d2ee6bb63e34d9bad59cdd4c20b0cd22d3753bd7a4510c2b71a956220"
Nov 04 00:16:59.909626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:59.909603    5074 scope.go:122] "RemoveContainer" containerID="1ec74d50dfd4e8376d1fd4332317468824b6d885a62fb1a981aa79ae243a858b"
Nov 04 00:16:59.920021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:16:59.920001    5074 scope.go:122] "RemoveContainer" containerID="c3a9f3736311604deba1722152900f752f0d510fcb56b215f5e8c1dcf6a4e76d"
Nov 04 00:17:10.457215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.457172    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"]
Nov 04 00:17:10.465955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.465925    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.478496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.478467    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-11301e\"/\"kube-root-ca.crt\""
Nov 04 00:17:10.482459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.480610    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-11301e\"/\"openshift-service-ca.crt\""
Nov 04 00:17:10.482459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.480617    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-11301e\"/\"default-dockercfg-sbtv5\""
Nov 04 00:17:10.482459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.481437    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-11301e\"/\"test-model-11301e-kserve-self-signed-certs\""
Nov 04 00:17:10.482459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.482262    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"]
Nov 04 00:17:10.572946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.572911    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.572946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.572952    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.573224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.572986    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.573224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.573131    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.573348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.573242    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.573348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.573312    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8p5b\" (UniqueName: \"kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.674472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.674426    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.675204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.675178    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.675415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.675391    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.675554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.675513    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.675661 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.675644    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.675777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.675752    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8p5b\" (UniqueName: \"kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.676158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.675114    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.676770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.676729    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.677182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.677158    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.681496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.681471    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.698771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.698653    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8p5b\" (UniqueName: \"kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.720854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.717198    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm\") pod \"test-model-11301e-kserve-6676d87b69-htplb\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") " pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:10.815324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:10.815025    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:11.113254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:11.113137    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"]
Nov 04 00:17:11.113632 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:11.113605    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f98430b_3385_4f3a_8d70_807e6b5b0825.slice/crio-43ffff5746c856f23ce087acb833480ffc5f0614d01c5c27dc169bcefff2ab18 WatchSource:0}: Error finding container 43ffff5746c856f23ce087acb833480ffc5f0614d01c5c27dc169bcefff2ab18: Status 404 returned error can't find the container with id 43ffff5746c856f23ce087acb833480ffc5f0614d01c5c27dc169bcefff2ab18
Nov 04 00:17:11.500982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:11.500941    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" event={"ID":"0f98430b-3385-4f3a-8d70-807e6b5b0825","Type":"ContainerStarted","Data":"7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f"}
Nov 04 00:17:11.501555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:11.500997    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" event={"ID":"0f98430b-3385-4f3a-8d70-807e6b5b0825","Type":"ContainerStarted","Data":"43ffff5746c856f23ce087acb833480ffc5f0614d01c5c27dc169bcefff2ab18"}
Nov 04 00:17:11.528638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:11.528576    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" podStartSLOduration=1.528521235 podStartE2EDuration="1.528521235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:11.525184756 +0000 UTC m=+2896.624784901" watchObservedRunningTime="2026-11-04 00:17:11.528521235 +0000 UTC m=+2896.628121379"
Nov 04 00:17:11.815353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:11.815268    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:20.815753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:20.815691    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:20.831084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:20.831059    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:21.625328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:21.625289    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:17:24.089879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.089836    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"]
Nov 04 00:17:24.096988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.096957    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.102676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.102647    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"]
Nov 04 00:17:24.106341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.106318    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-85ad28-dockercfg-8wvqk\""
Nov 04 00:17:24.122270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.122118    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-85ad28\""
Nov 04 00:17:24.142645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.142619    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"]
Nov 04 00:17:24.150918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.150880    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.151085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.151047    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsc6\" (UniqueName: \"kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6\") pod \"payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.156392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.156367    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.158194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.158176    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"]
Nov 04 00:17:24.265788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.264136    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsc6\" (UniqueName: \"kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6\") pod \"payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.265788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.264200    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxlt\" (UniqueName: \"kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt\") pod \"payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.265788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.264425    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.265788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.264506    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.265788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.265501    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.356663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.356549    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsc6\" (UniqueName: \"kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6\") pod \"payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.366372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.366214    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.366866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.366723    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxlt\" (UniqueName: \"kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt\") pod \"payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.368085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.368046    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.437760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.437643    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxlt\" (UniqueName: \"kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt\") pod \"payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.445293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.437942    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:24.665593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.665514    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:24.761380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.759052    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"]
Nov 04 00:17:24.764557 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:24.764525    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb489721_d8af_42cd_97df_49f2b24aa01c.slice/crio-1b9eb355af5a9c38f9e76c2fb07399d4ea029e7662ea2df16dfe8b4d67d8bce7 WatchSource:0}: Error finding container 1b9eb355af5a9c38f9e76c2fb07399d4ea029e7662ea2df16dfe8b4d67d8bce7: Status 404 returned error can't find the container with id 1b9eb355af5a9c38f9e76c2fb07399d4ea029e7662ea2df16dfe8b4d67d8bce7
Nov 04 00:17:24.980752 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:24.980711    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9a1d8f_0b21_4adb_9038_8b804224a320.slice/crio-76abd0c63c36478565ca3b407d617bfec40452b73c1af253355bf2171b0d2bc3 WatchSource:0}: Error finding container 76abd0c63c36478565ca3b407d617bfec40452b73c1af253355bf2171b0d2bc3: Status 404 returned error can't find the container with id 76abd0c63c36478565ca3b407d617bfec40452b73c1af253355bf2171b0d2bc3
Nov 04 00:17:24.981060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:24.981037    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"]
Nov 04 00:17:25.659096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.659060    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" event={"ID":"db489721-d8af-42cd-97df-49f2b24aa01c","Type":"ContainerStarted","Data":"0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1"}
Nov 04 00:17:25.659551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.659118    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" event={"ID":"db489721-d8af-42cd-97df-49f2b24aa01c","Type":"ContainerStarted","Data":"1b9eb355af5a9c38f9e76c2fb07399d4ea029e7662ea2df16dfe8b4d67d8bce7"}
Nov 04 00:17:25.659551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.659171    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:25.660988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.660962    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" event={"ID":"2c9a1d8f-0b21-4adb-9038-8b804224a320","Type":"ContainerStarted","Data":"f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd"}
Nov 04 00:17:25.661124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.660998    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" event={"ID":"2c9a1d8f-0b21-4adb-9038-8b804224a320","Type":"ContainerStarted","Data":"76abd0c63c36478565ca3b407d617bfec40452b73c1af253355bf2171b0d2bc3"}
Nov 04 00:17:25.661124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.661084    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:25.681616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:25.681559    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" podStartSLOduration=1.681542874 podStartE2EDuration="1.681542874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:25.68092715 +0000 UTC m=+2910.780527294" watchObservedRunningTime="2026-11-04 00:17:25.681542874 +0000 UTC m=+2910.781143019"
Nov 04 00:17:27.870753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.870677    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" podStartSLOduration=3.870662667 podStartE2EDuration="3.870662667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:25.7020744 +0000 UTC m=+2910.801674543" watchObservedRunningTime="2026-11-04 00:17:27.870662667 +0000 UTC m=+2912.970262811"
Nov 04 00:17:27.874499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.874470    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"]
Nov 04 00:17:27.879716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.879696    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:27.884426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.884377    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:17:27.886209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.886187    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:17:27.889314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.889291    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"]
Nov 04 00:17:27.922218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.922179    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-578r9\" (UniqueName: \"kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:27.922398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.922382    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:27.922493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:27.922477    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.024133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.024095    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.024316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.024198    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.024316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.024298    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-578r9\" (UniqueName: \"kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.025069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.025040    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.031758 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:28.026087    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:17:28.031758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.026668    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.036965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.036942    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-578r9\" (UniqueName: \"kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9\") pod \"maas-api-revoke-sub-6423ae74-tq67v\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.205948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.205915    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:28.512809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.506837    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"]
Nov 04 00:17:28.692256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:28.692217    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v" event={"ID":"74882772-cb33-45ea-96f8-018b402b034b","Type":"ContainerStarted","Data":"577d43f96e3e221f4d4c20f0343a2892942a895baea72753470b1f268f389c41"}
Nov 04 00:17:29.704085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:29.704045    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v" event={"ID":"74882772-cb33-45ea-96f8-018b402b034b","Type":"ContainerStarted","Data":"4a95a0ba596ba1dc8f142f639196159fe3c846ac93686e2011b932026ffe5522"}
Nov 04 00:17:29.729238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:29.729187    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v" podStartSLOduration=2.729172585 podStartE2EDuration="2.729172585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:29.728785771 +0000 UTC m=+2914.828385915" watchObservedRunningTime="2026-11-04 00:17:29.729172585 +0000 UTC m=+2914.828772731"
Nov 04 00:17:30.718166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:30.718123    5074 generic.go:356] "Generic (PLEG): container finished" podID="74882772-cb33-45ea-96f8-018b402b034b" containerID="4a95a0ba596ba1dc8f142f639196159fe3c846ac93686e2011b932026ffe5522" exitCode=0
Nov 04 00:17:30.718608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:30.718213    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v" event={"ID":"74882772-cb33-45ea-96f8-018b402b034b","Type":"ContainerDied","Data":"4a95a0ba596ba1dc8f142f639196159fe3c846ac93686e2011b932026ffe5522"}
Nov 04 00:17:31.952783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:31.951918    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:32.035589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.035492    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9\" (UniqueName: \"kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9\") pod \"74882772-cb33-45ea-96f8-018b402b034b\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") "
Nov 04 00:17:32.035589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.035578    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca\") pod \"74882772-cb33-45ea-96f8-018b402b034b\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") "
Nov 04 00:17:32.035855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.035707    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca\") pod \"74882772-cb33-45ea-96f8-018b402b034b\" (UID: \"74882772-cb33-45ea-96f8-018b402b034b\") "
Nov 04 00:17:32.038907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.038877    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca" pod "74882772-cb33-45ea-96f8-018b402b034b" (UID: "74882772-cb33-45ea-96f8-018b402b034b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:17:32.043157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.043129    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca" pod "74882772-cb33-45ea-96f8-018b402b034b" (UID: "74882772-cb33-45ea-96f8-018b402b034b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:17:32.049314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.049286    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9" pod "74882772-cb33-45ea-96f8-018b402b034b" (UID: "74882772-cb33-45ea-96f8-018b402b034b"). InnerVolumeSpecName "kube-api-access-578r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:32.137660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.137616    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-578r9\" (UniqueName: \"kubernetes.io/projected/74882772-cb33-45ea-96f8-018b402b034b-kube-api-access-578r9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:32.137660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.137659    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74882772-cb33-45ea-96f8-018b402b034b-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:32.137942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.137674    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74882772-cb33-45ea-96f8-018b402b034b-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:32.742649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.742615    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"
Nov 04 00:17:32.742649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.742628    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v" event={"ID":"74882772-cb33-45ea-96f8-018b402b034b","Type":"ContainerDied","Data":"577d43f96e3e221f4d4c20f0343a2892942a895baea72753470b1f268f389c41"}
Nov 04 00:17:32.742923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:32.742720    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="577d43f96e3e221f4d4c20f0343a2892942a895baea72753470b1f268f389c41"
Nov 04 00:17:36.399356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.399137    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"]
Nov 04 00:17:36.400212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.400192    5074 cpu_manager.go:414] "containerMap: removing container" podUID="74882772-cb33-45ea-96f8-018b402b034b" containerName="revoke-keys"
Nov 04 00:17:36.400212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.400212    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74882772-cb33-45ea-96f8-018b402b034b" containerName="revoke-keys"
Nov 04 00:17:36.400432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.400417    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="74882772-cb33-45ea-96f8-018b402b034b" containerName="revoke-keys"
Nov 04 00:17:36.408431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.408401    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.414427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.414402    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"]
Nov 04 00:17:36.415031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.414999    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a7cbc3-dockercfg-q8rj8\""
Nov 04 00:17:36.415167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.415063    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a7cbc3\""
Nov 04 00:17:36.466070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.466031    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"]
Nov 04 00:17:36.477007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.476972    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.482038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.481999    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"]
Nov 04 00:17:36.503718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.503685    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2t9\" (UniqueName: \"kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9\") pod \"payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.503968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.503942    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.616761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.607769    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2t9\" (UniqueName: \"kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9\") pod \"payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.616761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.607951    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.616761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.608000    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.616761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.608086    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfkl\" (UniqueName: \"kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl\") pod \"payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.616761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.609555    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.637762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.637564    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2t9\" (UniqueName: \"kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9\") pod \"payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.679226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.677655    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:17:36.689138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.681418    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:17:36.719705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.710393    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfkl\" (UniqueName: \"kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl\") pod \"payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.719705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.710750    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.719705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.711850    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.752383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.746961    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfkl\" (UniqueName: \"kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl\") pod \"payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:36.758470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.758437    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:36.794971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:36.794663    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:37.082321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.082291    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"]
Nov 04 00:17:37.083046 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:37.083023    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode560ec02_fac5_45d3_bfd7_c9c6291bf776.slice/crio-e096e121c125ffc839f81c9d54c7fa01a9578adb9e283a95628a84e60f90929b WatchSource:0}: Error finding container e096e121c125ffc839f81c9d54c7fa01a9578adb9e283a95628a84e60f90929b: Status 404 returned error can't find the container with id e096e121c125ffc839f81c9d54c7fa01a9578adb9e283a95628a84e60f90929b
Nov 04 00:17:37.131072 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.130663    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"]
Nov 04 00:17:37.133702 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:37.133263    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03935378_bfbf_4b4e_a95a_13b693cab31c.slice/crio-471ee90bfcf4edbd5c911ada75e58ad55156a168af0b47f4ea2d26c71c5641b6 WatchSource:0}: Error finding container 471ee90bfcf4edbd5c911ada75e58ad55156a168af0b47f4ea2d26c71c5641b6: Status 404 returned error can't find the container with id 471ee90bfcf4edbd5c911ada75e58ad55156a168af0b47f4ea2d26c71c5641b6
Nov 04 00:17:37.797367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.797332    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5" event={"ID":"e560ec02-fac5-45d3-bfd7-c9c6291bf776","Type":"ContainerStarted","Data":"8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8"}
Nov 04 00:17:37.797866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.797399    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:37.797866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.797420    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5" event={"ID":"e560ec02-fac5-45d3-bfd7-c9c6291bf776","Type":"ContainerStarted","Data":"e096e121c125ffc839f81c9d54c7fa01a9578adb9e283a95628a84e60f90929b"}
Nov 04 00:17:37.799314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.799283    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" event={"ID":"03935378-bfbf-4b4e-a95a-13b693cab31c","Type":"ContainerStarted","Data":"8379205087a9ddb46881de60cef4c12f12cf9eb5d3d4605c23067123d2a30bdf"}
Nov 04 00:17:37.799440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.799330    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" event={"ID":"03935378-bfbf-4b4e-a95a-13b693cab31c","Type":"ContainerStarted","Data":"471ee90bfcf4edbd5c911ada75e58ad55156a168af0b47f4ea2d26c71c5641b6"}
Nov 04 00:17:37.799440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.799425    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:37.824266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.824207    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5" podStartSLOduration=1.824190195 podStartE2EDuration="1.824190195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:37.816416437 +0000 UTC m=+2922.916016581" watchObservedRunningTime="2026-11-04 00:17:37.824190195 +0000 UTC m=+2922.923790339"
Nov 04 00:17:37.839450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:37.839398    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" podStartSLOduration=1.839383866 podStartE2EDuration="1.839383866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:37.835623119 +0000 UTC m=+2922.935223272" watchObservedRunningTime="2026-11-04 00:17:37.839383866 +0000 UTC m=+2922.938984019"
Nov 04 00:17:38.408965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.408926    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"]
Nov 04 00:17:38.428216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.428185    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.429795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.428299    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"]
Nov 04 00:17:38.436749 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.436704    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:17:38.436913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.436853    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:17:38.455795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.455026    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.455795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.455084    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84l5\" (UniqueName: \"kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.455795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.455121    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.556676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.556626    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.556895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.556697    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84l5\" (UniqueName: \"kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.556895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.556758    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.557618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.557591    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.559126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.559102    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.569697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.569672    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84l5\" (UniqueName: \"kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5\") pod \"maas-api-revoke-sub-58e44247-nhrdn\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:38.756965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:38.756927    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:39.086776 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:39.086726    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc372e3a3_2e12_49b2_bbba_c8d15319ce32.slice/crio-9dcbfc97a025438cf2ae78a6033ced996410a92acdc8f317a44ee9bb683387c0 WatchSource:0}: Error finding container 9dcbfc97a025438cf2ae78a6033ced996410a92acdc8f317a44ee9bb683387c0: Status 404 returned error can't find the container with id 9dcbfc97a025438cf2ae78a6033ced996410a92acdc8f317a44ee9bb683387c0
Nov 04 00:17:39.087213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:39.087189    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"]
Nov 04 00:17:39.831364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:39.831262    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn" event={"ID":"c372e3a3-2e12-49b2-bbba-c8d15319ce32","Type":"ContainerStarted","Data":"6c912e5e3cfa25a0a44120ef751dad3fcfe88d5bcc622b9186c78081b8aa6ef8"}
Nov 04 00:17:39.831364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:39.831320    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn" event={"ID":"c372e3a3-2e12-49b2-bbba-c8d15319ce32","Type":"ContainerStarted","Data":"9dcbfc97a025438cf2ae78a6033ced996410a92acdc8f317a44ee9bb683387c0"}
Nov 04 00:17:39.853980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:39.853930    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn" podStartSLOduration=1.853915765 podStartE2EDuration="1.853915765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:39.850295412 +0000 UTC m=+2924.949895572" watchObservedRunningTime="2026-11-04 00:17:39.853915765 +0000 UTC m=+2924.953515912"
Nov 04 00:17:40.843823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:40.843782    5074 generic.go:356] "Generic (PLEG): container finished" podID="c372e3a3-2e12-49b2-bbba-c8d15319ce32" containerID="6c912e5e3cfa25a0a44120ef751dad3fcfe88d5bcc622b9186c78081b8aa6ef8" exitCode=0
Nov 04 00:17:40.844246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:40.843869    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn" event={"ID":"c372e3a3-2e12-49b2-bbba-c8d15319ce32","Type":"ContainerDied","Data":"6c912e5e3cfa25a0a44120ef751dad3fcfe88d5bcc622b9186c78081b8aa6ef8"}
Nov 04 00:17:42.064993 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.064965    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:42.114886 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.114799    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca\") pod \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") "
Nov 04 00:17:42.115062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.114896    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca\") pod \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") "
Nov 04 00:17:42.115062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.114984    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5\" (UniqueName: \"kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5\") pod \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\" (UID: \"c372e3a3-2e12-49b2-bbba-c8d15319ce32\") "
Nov 04 00:17:42.115608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.115577    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca" pod "c372e3a3-2e12-49b2-bbba-c8d15319ce32" (UID: "c372e3a3-2e12-49b2-bbba-c8d15319ce32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:17:42.116564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.116532    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:42.118282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.118245    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca" pod "c372e3a3-2e12-49b2-bbba-c8d15319ce32" (UID: "c372e3a3-2e12-49b2-bbba-c8d15319ce32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:17:42.120032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.119997    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5" pod "c372e3a3-2e12-49b2-bbba-c8d15319ce32" (UID: "c372e3a3-2e12-49b2-bbba-c8d15319ce32"). InnerVolumeSpecName "kube-api-access-f84l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:42.218198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.218163    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c372e3a3-2e12-49b2-bbba-c8d15319ce32-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:42.218198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.218197    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84l5\" (UniqueName: \"kubernetes.io/projected/c372e3a3-2e12-49b2-bbba-c8d15319ce32-kube-api-access-f84l5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:42.864482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.864449    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"
Nov 04 00:17:42.864689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.864483    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn" event={"ID":"c372e3a3-2e12-49b2-bbba-c8d15319ce32","Type":"ContainerDied","Data":"9dcbfc97a025438cf2ae78a6033ced996410a92acdc8f317a44ee9bb683387c0"}
Nov 04 00:17:42.864689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:42.864534    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dcbfc97a025438cf2ae78a6033ced996410a92acdc8f317a44ee9bb683387c0"
Nov 04 00:17:48.822079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:48.822041    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:17:48.822504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:48.822123    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:17:50.442833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.442788    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"]
Nov 04 00:17:50.444262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.444229    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c372e3a3-2e12-49b2-bbba-c8d15319ce32" containerName="revoke-keys"
Nov 04 00:17:50.444262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.444262    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c372e3a3-2e12-49b2-bbba-c8d15319ce32" containerName="revoke-keys"
Nov 04 00:17:50.444559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.444538    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c372e3a3-2e12-49b2-bbba-c8d15319ce32" containerName="revoke-keys"
Nov 04 00:17:50.471821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.471786    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"]
Nov 04 00:17:50.472152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.472135    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.495214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.495182    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\""
Nov 04 00:17:50.495566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.495508    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-559c901bbcfd\""
Nov 04 00:17:50.500132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.497515    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\""
Nov 04 00:17:50.500132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.497634    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\""
Nov 04 00:17:50.500132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.497654    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\""
Nov 04 00:17:50.500132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.497776    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-d6htd\""
Nov 04 00:17:50.565064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.565025    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.565386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.565361    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.565518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.565503    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.565837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.565800    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsbh\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.667135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.667095    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.667307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.667150    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.667361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.667346    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsbh\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.667523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.667506    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.668075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.668041    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.668318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.668289    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.670304 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.670274    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.683482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.683450    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsbh\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.872055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.871959    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:50.878832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.878799    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"]
Nov 04 00:17:50.930436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:50.930394    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" podUID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" containerName="sim" containerID="cri-o://bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45" gracePeriod=30
Nov 04 00:17:51.068075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.068033    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" podUID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" containerName="sim" probeResult="failure" output="Get \"https://10.132.1.57:8000/ready\": dial tcp 10.132.1.57:8000: connect: connection refused"
Nov 04 00:17:51.183340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.180006    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"]
Nov 04 00:17:51.212910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.212874    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"]
Nov 04 00:17:51.213081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.213009    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.218865 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.218830    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a002b\""
Nov 04 00:17:51.219046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.218948    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"agc-cred-14b358f9ee72\""
Nov 04 00:17:51.219125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.219053    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-e2e--7c418871-dockercfg-7sffv\""
Nov 04 00:17:51.219185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.219141    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"routing-overlay\""
Nov 04 00:17:51.219245 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.219198    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"kube-root-ca.crt\""
Nov 04 00:17:51.219308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.219275    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"openshift-service-ca.crt\""
Nov 04 00:17:51.283395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.283357    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"]
Nov 04 00:17:51.284176 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:51.284148    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e88872_3595_471b_bdaf_2646f3862bec.slice/crio-db85c086bc156423a040cc2639933d5663c2080f951baaa732862f835db0982f WatchSource:0}: Error finding container db85c086bc156423a040cc2639933d5663c2080f951baaa732862f835db0982f: Status 404 returned error can't find the container with id db85c086bc156423a040cc2639933d5663c2080f951baaa732862f835db0982f
Nov 04 00:17:51.291712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.291680    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.291846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.291730    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbgh\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.291921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.291899    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.292176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.292155    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.322641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.322617    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:17:51.393145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.393107    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.393342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.393161    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbgh\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.393342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.393216    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.393606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.393469    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.394192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.394150    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.394528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.394476    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.400349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.400313    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.437856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.425564    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbgh\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.495127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.495087    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls\" (UniqueName: \"kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls\") pod \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") "
Nov 04 00:17:51.495638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.495173    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45\" (UniqueName: \"kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45\") pod \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\" (UID: \"9e15edf7-bded-4214-9a4d-097e9f44b4ed\") "
Nov 04 00:17:51.497562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.497531    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls" pod "9e15edf7-bded-4214-9a4d-097e9f44b4ed" (UID: "9e15edf7-bded-4214-9a4d-097e9f44b4ed"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:17:51.497690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.497576    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45" pod "9e15edf7-bded-4214-9a4d-097e9f44b4ed" (UID: "9e15edf7-bded-4214-9a4d-097e9f44b4ed"). InnerVolumeSpecName "kube-api-access-xwr45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:51.539911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.539875    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:51.598130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.598056    5074 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9e15edf7-bded-4214-9a4d-097e9f44b4ed-tls\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:51.598130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.598101    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwr45\" (UniqueName: \"kubernetes.io/projected/9e15edf7-bded-4214-9a4d-097e9f44b4ed-kube-api-access-xwr45\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:51.829270 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:17:51.829238    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41cc5c5_5bba_4b48_93ef_970b546b9a16.slice/crio-2e3a9ee8adc9f7e64712e0a61c09228ea04dfc0274d05c1edcdfc8912e3187ea WatchSource:0}: Error finding container 2e3a9ee8adc9f7e64712e0a61c09228ea04dfc0274d05c1edcdfc8912e3187ea: Status 404 returned error can't find the container with id 2e3a9ee8adc9f7e64712e0a61c09228ea04dfc0274d05c1edcdfc8912e3187ea
Nov 04 00:17:51.831717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:51.831688    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"]
Nov 04 00:17:52.013479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.013386    5074 generic.go:356] "Generic (PLEG): container finished" podID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" containerID="bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45" exitCode=0
Nov 04 00:17:52.013479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.013428    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"
Nov 04 00:17:52.013703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.013470    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" event={"ID":"9e15edf7-bded-4214-9a4d-097e9f44b4ed","Type":"ContainerDied","Data":"bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45"}
Nov 04 00:17:52.013703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.013522    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq" event={"ID":"9e15edf7-bded-4214-9a4d-097e9f44b4ed","Type":"ContainerDied","Data":"2a819afed4ee52815ac763477443d56acb933838c853466fbd0f00e0b787bded"}
Nov 04 00:17:52.013703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.013547    5074 scope.go:122] "RemoveContainer" containerID="bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45"
Nov 04 00:17:52.016471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.016442    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" event={"ID":"b41cc5c5-5bba-4b48-93ef-970b546b9a16","Type":"ContainerStarted","Data":"a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133"}
Nov 04 00:17:52.016614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.016494    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" event={"ID":"b41cc5c5-5bba-4b48-93ef-970b546b9a16","Type":"ContainerStarted","Data":"2e3a9ee8adc9f7e64712e0a61c09228ea04dfc0274d05c1edcdfc8912e3187ea"}
Nov 04 00:17:52.016952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.016929    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:52.019420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.019392    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" event={"ID":"d5e88872-3595-471b-bdaf-2646f3862bec","Type":"ContainerStarted","Data":"f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9"}
Nov 04 00:17:52.019558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.019438    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" event={"ID":"d5e88872-3595-471b-bdaf-2646f3862bec","Type":"ContainerStarted","Data":"db85c086bc156423a040cc2639933d5663c2080f951baaa732862f835db0982f"}
Nov 04 00:17:52.019802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.019730    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:52.028581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.028559    5074 scope.go:122] "RemoveContainer" containerID="bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45"
Nov 04 00:17:52.028883 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:52.028864    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45\": container with ID starting with bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45 not found: ID does not exist" containerID="bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45"
Nov 04 00:17:52.028959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.028889    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45"} err="failed to get container status \"bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45\": rpc error: code = NotFound desc = could not find container \"bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45\": container with ID starting with bcbbe01ab7c54fba933242a84e7418a07d6cb1a708f4da1c91c965ed418f9e45 not found: ID does not exist"
Nov 04 00:17:52.040392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.040342    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" podStartSLOduration=1.040331353 podStartE2EDuration="1.040331353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:52.038958614 +0000 UTC m=+2937.138558759" watchObservedRunningTime="2026-11-04 00:17:52.040331353 +0000 UTC m=+2937.139931497"
Nov 04 00:17:52.062807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.062781    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"]
Nov 04 00:17:52.070992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.070963    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-241087/e2e-usage-sim-10ac07-6c4b9c78f-58clq"]
Nov 04 00:17:52.089211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.089161    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" podStartSLOduration=2.089148052 podStartE2EDuration="2.089148052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:52.084598704 +0000 UTC m=+2937.184198849" watchObservedRunningTime="2026-11-04 00:17:52.089148052 +0000 UTC m=+2937.188748195"
Nov 04 00:17:52.472932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.472891    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"]
Nov 04 00:17:52.973491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:52.973452    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"]
Nov 04 00:17:53.023156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:53.023118    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" path="/var/lib/kubelet/pods/9e15edf7-bded-4214-9a4d-097e9f44b4ed/volumes"
Nov 04 00:17:53.034560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:53.034532    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" secret="" err="secret \"payload-processing-external-model-e2e--0bada0ae-dockercfg-d6htd\" not found"
Nov 04 00:17:53.034775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:53.034735    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" secret="" err="secret \"payload-processing-external-model-e2e--7c418871-dockercfg-7sffv\" not found"
Nov 04 00:17:53.078804 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.078772    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:53.078993 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.078837    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:17:53.578822374 +0000 UTC m=+2938.678422527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:53.078993 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.078977    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-plugins-e2e-extmod-b-1b5fca: configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:53.079133 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079053    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5fca/agc-cred-559c901bbcfd: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:53.079133 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079068    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:53.079133 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079055    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:17:53.579035224 +0000 UTC m=+2938.678635346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:53.079133 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079124    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:17:53.579106965 +0000 UTC m=+2938.678707104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:53.079377 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079361    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-14b358f9ee72: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:53.079425 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079380    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:53.079463 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.079430    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:17:53.579416981 +0000 UTC m=+2938.679017116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:53.589780 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.589726    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-plugins-e2e-extmod-b-1b5fca: configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:53.589982 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.589836    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:17:54.589814504 +0000 UTC m=+2939.689414646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:53.589982 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.589848    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5fca/agc-cred-559c901bbcfd: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:53.589982 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.589864    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:53.589982 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.589904    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:17:54.589893903 +0000 UTC m=+2939.689494039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:53.590220 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.590047    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-14b358f9ee72: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:53.590220 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.590061    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:53.590220 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.590096    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:17:54.590084622 +0000 UTC m=+2939.689684766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:53.590220 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.590140    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:53.590220 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:53.590208    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:17:54.590190657 +0000 UTC m=+2939.689790794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:54.611064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611030    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-14b358f9ee72: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:54.611064 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611060    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611082    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611121    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:17:56.611103591 +0000 UTC m=+2941.710703713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611154    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-plugins-e2e-extmod-b-1b5fca: configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611209    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:17:56.61119471 +0000 UTC m=+2941.710794845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611233    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:17:56.611224148 +0000 UTC m=+2941.710824269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611257    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5fca/agc-cred-559c901bbcfd: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611268    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:54.611581 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:54.611301    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:17:56.611290348 +0000 UTC m=+2941.710890471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:56.649629 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649594    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-plugins-e2e-extmod-b-1b5fca: configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649666    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:18:00.649648125 +0000 UTC m=+2945.749248249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-1b5fca" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649679    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5fca/agc-cred-559c901bbcfd: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649700    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz: secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649728    5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-14b358f9ee72: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649764    5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7: secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649756    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials podName:d5e88872-3595-471b-bdaf-2646f3862bec nodeName:}" failed. No retries permitted until 2026-11-04 00:18:00.649730206 +0000 UTC m=+2945.749330328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec") : secret "agc-cred-559c901bbcfd" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649797    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:18:00.649788626 +0000 UTC m=+2945.749388778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : secret "agc-cred-14b358f9ee72" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649813    5074 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:56.650068 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:56.649838    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume podName:b41cc5c5-5bba-4b48-93ef-970b546b9a16 nodeName:}" failed. No retries permitted until 2026-11-04 00:18:00.649831456 +0000 UTC m=+2945.749431631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found
Nov 04 00:17:58.032262 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:17:58.032220    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:17:59.047057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.047009    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" podUID="b41cc5c5-5bba-4b48-93ef-970b546b9a16" containerName="payload-processing" containerID="cri-o://a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133" gracePeriod=25
Nov 04 00:17:59.047506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.047020    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" podUID="d5e88872-3595-471b-bdaf-2646f3862bec" containerName="payload-processing" containerID="cri-o://f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9" gracePeriod=25
Nov 04 00:17:59.391243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.391217    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:17:59.395269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.395244    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:17:59.496697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.496660    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials\") pod \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") "
Nov 04 00:17:59.496903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.496816    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh\") pod \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") "
Nov 04 00:17:59.496978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.496925    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume\") pod \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") "
Nov 04 00:17:59.497042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497013    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume\") pod \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\" (UID: \"b41cc5c5-5bba-4b48-93ef-970b546b9a16\") "
Nov 04 00:17:59.497102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497071    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials\") pod \"d5e88872-3595-471b-bdaf-2646f3862bec\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") "
Nov 04 00:17:59.497204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497172    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh\") pod \"d5e88872-3595-471b-bdaf-2646f3862bec\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") "
Nov 04 00:17:59.497350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497327    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume" pod "b41cc5c5-5bba-4b48-93ef-970b546b9a16" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:17:59.497409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497353    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume\") pod \"d5e88872-3595-471b-bdaf-2646f3862bec\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") "
Nov 04 00:17:59.497474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497414    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume" pod "b41cc5c5-5bba-4b48-93ef-970b546b9a16" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:17:59.497474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497423    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume\") pod \"d5e88872-3595-471b-bdaf-2646f3862bec\" (UID: \"d5e88872-3595-471b-bdaf-2646f3862bec\") "
Nov 04 00:17:59.497759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497720    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume" pod "d5e88872-3595-471b-bdaf-2646f3862bec" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:17:59.497860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.497784    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume" pod "d5e88872-3595-471b-bdaf-2646f3862bec" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:17:59.498340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.498309    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.498468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.498342    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41cc5c5-5bba-4b48-93ef-970b546b9a16-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.498468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.498360    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.498468 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.498375    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e88872-3595-471b-bdaf-2646f3862bec-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.499377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.499350    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials" pod "b41cc5c5-5bba-4b48-93ef-970b546b9a16" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:59.499586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.499540    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh" pod "b41cc5c5-5bba-4b48-93ef-970b546b9a16" (UID: "b41cc5c5-5bba-4b48-93ef-970b546b9a16"). InnerVolumeSpecName "kube-api-access-7xbgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:59.499681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.499625    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh" pod "d5e88872-3595-471b-bdaf-2646f3862bec" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec"). InnerVolumeSpecName "kube-api-access-mbsbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:59.499681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.499653    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials" pod "d5e88872-3595-471b-bdaf-2646f3862bec" (UID: "d5e88872-3595-471b-bdaf-2646f3862bec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:17:59.599991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.599893    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.599991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.599927    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbgh\" (UniqueName: \"kubernetes.io/projected/b41cc5c5-5bba-4b48-93ef-970b546b9a16-kube-api-access-7xbgh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.599991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.599939    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:17:59.599991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:17:59.599950    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsbh\" (UniqueName: \"kubernetes.io/projected/d5e88872-3595-471b-bdaf-2646f3862bec-kube-api-access-mbsbh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:00.020979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.020936    5074 scope.go:122] "RemoveContainer" containerID="bf68460b537ad4bff32cbef2b1a8a04fe205562698a2a8db48a251ed0fd4e1de"
Nov 04 00:18:00.034969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.034942    5074 scope.go:122] "RemoveContainer" containerID="d889b965ee2e7fc648adcc77f3fbe0d1289e2aa1c5d35da2b80ea3a41ff63fbb"
Nov 04 00:18:00.049132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.048958    5074 scope.go:122] "RemoveContainer" containerID="fe08c857a328e9ab0871c5cb5cc4cdb2f5e08faf6f646c8522223673f2af4c25"
Nov 04 00:18:00.060667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.060639    5074 scope.go:122] "RemoveContainer" containerID="4db58e06a8ff835d7f2284bb39dac06f51080ceed61c679079f9cb923bd9377e"
Nov 04 00:18:00.088801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.088774    5074 scope.go:122] "RemoveContainer" containerID="76ec891129393e4bd6f4f2e291eab26d21d182cdd77bcb2a84457622f651e580"
Nov 04 00:18:00.135320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.135290    5074 generic.go:356] "Generic (PLEG): container finished" podID="b41cc5c5-5bba-4b48-93ef-970b546b9a16" containerID="a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133" exitCode=0
Nov 04 00:18:00.135455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.135334    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"
Nov 04 00:18:00.135455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.135362    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" event={"ID":"b41cc5c5-5bba-4b48-93ef-970b546b9a16","Type":"ContainerDied","Data":"a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133"}
Nov 04 00:18:00.135455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.135438    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7" event={"ID":"b41cc5c5-5bba-4b48-93ef-970b546b9a16","Type":"ContainerDied","Data":"2e3a9ee8adc9f7e64712e0a61c09228ea04dfc0274d05c1edcdfc8912e3187ea"}
Nov 04 00:18:00.135455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.135454    5074 scope.go:122] "RemoveContainer" containerID="a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133"
Nov 04 00:18:00.141353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.141324    5074 generic.go:356] "Generic (PLEG): container finished" podID="d5e88872-3595-471b-bdaf-2646f3862bec" containerID="f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9" exitCode=0
Nov 04 00:18:00.141640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.141390    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"
Nov 04 00:18:00.141640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.141412    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" event={"ID":"d5e88872-3595-471b-bdaf-2646f3862bec","Type":"ContainerDied","Data":"f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9"}
Nov 04 00:18:00.141640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.141471    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz" event={"ID":"d5e88872-3595-471b-bdaf-2646f3862bec","Type":"ContainerDied","Data":"db85c086bc156423a040cc2639933d5663c2080f951baaa732862f835db0982f"}
Nov 04 00:18:00.155665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.155626    5074 scope.go:122] "RemoveContainer" containerID="a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133"
Nov 04 00:18:00.155937 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:18:00.155919    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133\": container with ID starting with a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133 not found: ID does not exist" containerID="a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133"
Nov 04 00:18:00.155992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.155945    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133"} err="failed to get container status \"a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133\": rpc error: code = NotFound desc = could not find container \"a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133\": container with ID starting with a6b90db6811f43a53439e9658ccf266dd8bc6ba5b460f73bf873272b3c1fd133 not found: ID does not exist"
Nov 04 00:18:00.155992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.155960    5074 scope.go:122] "RemoveContainer" containerID="f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9"
Nov 04 00:18:00.168287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.168226    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"]
Nov 04 00:18:00.168370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.168337    5074 scope.go:122] "RemoveContainer" containerID="f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9"
Nov 04 00:18:00.168680 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:18:00.168660    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9\": container with ID starting with f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9 not found: ID does not exist" containerID="f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9"
Nov 04 00:18:00.168764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.168693    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9"} err="failed to get container status \"f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9\": rpc error: code = NotFound desc = could not find container \"f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9\": container with ID starting with f3cd054886cc965c189b5d54db635e228424600c65d5fe88304958132195a6a9 not found: ID does not exist"
Nov 04 00:18:00.177171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.177135    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-84447tfm7"]
Nov 04 00:18:00.202277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.202230    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"]
Nov 04 00:18:00.210289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:00.210243    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-8559k8jmz"]
Nov 04 00:18:01.035367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:01.035323    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41cc5c5-5bba-4b48-93ef-970b546b9a16" path="/var/lib/kubelet/pods/b41cc5c5-5bba-4b48-93ef-970b546b9a16/volumes"
Nov 04 00:18:01.036009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:01.035983    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e88872-3595-471b-bdaf-2646f3862bec" path="/var/lib/kubelet/pods/d5e88872-3595-471b-bdaf-2646f3862bec/volumes"
Nov 04 00:18:18.944798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.944711    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"]
Nov 04 00:18:18.946311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946282    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" containerName="sim"
Nov 04 00:18:18.946311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946310    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" containerName="sim"
Nov 04 00:18:18.946555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946354    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d5e88872-3595-471b-bdaf-2646f3862bec" containerName="payload-processing"
Nov 04 00:18:18.946555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946362    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e88872-3595-471b-bdaf-2646f3862bec" containerName="payload-processing"
Nov 04 00:18:18.946555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946389    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b41cc5c5-5bba-4b48-93ef-970b546b9a16" containerName="payload-processing"
Nov 04 00:18:18.946555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946399    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41cc5c5-5bba-4b48-93ef-970b546b9a16" containerName="payload-processing"
Nov 04 00:18:18.946799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946638    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e15edf7-bded-4214-9a4d-097e9f44b4ed" containerName="sim"
Nov 04 00:18:18.946799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946659    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e88872-3595-471b-bdaf-2646f3862bec" containerName="payload-processing"
Nov 04 00:18:18.946799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.946672    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41cc5c5-5bba-4b48-93ef-970b546b9a16" containerName="payload-processing"
Nov 04 00:18:18.953708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.953677    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:18.981344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:18.981314    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"]
Nov 04 00:18:19.031937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.031902    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.032301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.032277    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.032459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.032444    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.032620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.032592    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr9h\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.032728 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.032714    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.032892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.032877    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.033009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.032995    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.033171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.033157    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.033305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.033292    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.056856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.056813    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8cc1c0-openshift-default-dockercfg-nxf65\""
Nov 04 00:18:19.135106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135068    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.135292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135115    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.135292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135196    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.135417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135289    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135635    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135649    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135826    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135892    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135945    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135967    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr9h\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.135995    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.136212    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.136351    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.136627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.136468    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.137894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.137874    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.139027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.139007    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.148713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.148686    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.148869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.148797    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr9h\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.273778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.273439    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:19.585264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.585231    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"]
Nov 04 00:18:19.586990 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:18:19.586957    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f62472d_3985_44e3_ade7_a2a87d28ffcb.slice/crio-5fcf091c994136bd0c2be3711473a2a9d1d106b925de3b1983ebd3f06601641b WatchSource:0}: Error finding container 5fcf091c994136bd0c2be3711473a2a9d1d106b925de3b1983ebd3f06601641b: Status 404 returned error can't find the container with id 5fcf091c994136bd0c2be3711473a2a9d1d106b925de3b1983ebd3f06601641b
Nov 04 00:18:19.590522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.590483    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:19.590615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.590568    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:19.590659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:19.590612    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:20.358260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:20.358222    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" event={"ID":"9f62472d-3985-44e3-ade7-a2a87d28ffcb","Type":"ContainerStarted","Data":"25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508"}
Nov 04 00:18:20.358796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:20.358277    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" event={"ID":"9f62472d-3985-44e3-ade7-a2a87d28ffcb","Type":"ContainerStarted","Data":"5fcf091c994136bd0c2be3711473a2a9d1d106b925de3b1983ebd3f06601641b"}
Nov 04 00:18:20.387565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:20.387497    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podStartSLOduration=2.387481458 podStartE2EDuration="2.387481458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:20.384865117 +0000 UTC m=+2965.484465262" watchObservedRunningTime="2026-11-04 00:18:20.387481458 +0000 UTC m=+2965.487081603"
Nov 04 00:18:21.273735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:21.273696    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:21.275522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:21.275492    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 04 00:18:21.275654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:21.275556    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 04 00:18:22.043560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:22.043145    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"]
Nov 04 00:18:22.142555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:22.142457    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e135e17-c998q"]
Nov 04 00:18:22.274448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:22.274416    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 04 00:18:22.274619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:22.274487    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 04 00:18:23.045932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:23.045892    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23d47eb-235c-4547-b6ba-fc5b6059094a" path="/var/lib/kubelet/pods/f23d47eb-235c-4547-b6ba-fc5b6059094a/volumes"
Nov 04 00:18:23.274412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:23.274376    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 04 00:18:23.274588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:23.274443    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 04 00:18:24.274131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:24.274096    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 04 00:18:24.274544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:24.274152    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 04 00:18:25.274056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:25.274013    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 04 00:18:25.274247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:25.274074    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 04 00:18:26.274848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:26.274804    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 04 00:18:26.275358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:26.274881    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 04 00:18:27.280133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:27.280102    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:27.280872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:27.280847    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:27.281666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:27.281644    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:18:28.050990 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:18:28.050946    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:18:30.608235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.608201    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"]
Nov 04 00:18:30.655225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.655188    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"]
Nov 04 00:18:30.655434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.655208    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.670403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.667546    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:18:30.670403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.667661    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:18:30.775325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.775279    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.775512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.775394    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb5ps\" (UniqueName: \"kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.775512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.775500    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.877374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.877289    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.877571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.877550    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.877656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.877641    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb5ps\" (UniqueName: \"kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.878039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.878016    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.880279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.880254    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.890985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.890954    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb5ps\" (UniqueName: \"kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps\") pod \"maas-api-revoke-sub-7956f4e8-wbvjg\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:30.993832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:30.993791    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:31.312426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:31.312392    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"]
Nov 04 00:18:31.564811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:31.564302    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg" event={"ID":"55e17594-e4b1-4c74-9638-7dd2509a0e64","Type":"ContainerStarted","Data":"f53e0c7b5a4a0778f557e8af24c9ff93b6b6c89bade8623654ea926d41be49a8"}
Nov 04 00:18:32.639028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.638971    5074 generic.go:356] "Generic (PLEG): container finished" podID="55e17594-e4b1-4c74-9638-7dd2509a0e64" containerID="d1504a31a04c8de25a042b24e6887f6114177bb817e5fe54b9613a8483ec6c4b" exitCode=0
Nov 04 00:18:32.639861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.639830    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg" event={"ID":"55e17594-e4b1-4c74-9638-7dd2509a0e64","Type":"ContainerDied","Data":"d1504a31a04c8de25a042b24e6887f6114177bb817e5fe54b9613a8483ec6c4b"}
Nov 04 00:18:32.774557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.774473    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"]
Nov 04 00:18:32.788672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.787909    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"]
Nov 04 00:18:32.803761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.788947    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.831978    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832063    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832093    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832122    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832153    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832234    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832327    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832342    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.832558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.832361    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4fw\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.864350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.864317    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-85ad28-openshift-default-dockercfg-5xfzh\""
Nov 04 00:18:32.930186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.930145    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"]
Nov 04 00:18:32.938442 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.938412    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:32.940288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940255    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940295    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940326    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4fw\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940389    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940470    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940515    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940557    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.940671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940635    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.941032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.940800    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.941324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.941304    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.942675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.942645    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.942818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.942780    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.943702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.943032    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.943702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.943657    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.946776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.946419    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"]
Nov 04 00:18:32.946776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.946513    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.958779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.958381    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.961800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.961684    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.964452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.964429    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4fw\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:32.994009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:32.993967    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"]
Nov 04 00:18:33.005681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.005650    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"]
Nov 04 00:18:33.005837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.005822    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.020309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.020050    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d94e94-dockercfg-h2476\""
Nov 04 00:18:33.020991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.020948    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d94e94\""
Nov 04 00:18:33.049832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.047628    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.049832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.047860    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2km7f\" (UniqueName: \"kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f\") pod \"payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.152801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.152756    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2km7f\" (UniqueName: \"kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f\") pod \"payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.153090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.153067    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q5qk\" (UniqueName: \"kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk\") pod \"payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.153274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.153255    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.153532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.153514    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.154890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.154684    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.176057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.176010    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:33.185295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.185257    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2km7f\" (UniqueName: \"kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f\") pod \"payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.259251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.259205    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q5qk\" (UniqueName: \"kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk\") pod \"payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.259573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.259553    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.266805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.266770    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.310901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.310721    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q5qk\" (UniqueName: \"kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk\") pod \"payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.349084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.349034    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:33.356540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.353326    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:33.636602 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:18:33.636564    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcc76b8_d66d_4fc7_97f7_a8842c5f8520.slice/crio-71520ebe61a3f5896542ee0e9248d9ad5ffc298a00f81cab2240123dd4a5e91a WatchSource:0}: Error finding container 71520ebe61a3f5896542ee0e9248d9ad5ffc298a00f81cab2240123dd4a5e91a: Status 404 returned error can't find the container with id 71520ebe61a3f5896542ee0e9248d9ad5ffc298a00f81cab2240123dd4a5e91a
Nov 04 00:18:33.664463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.662187    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:33.664463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.662306    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:33.664463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.662366    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:33.689799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.689768    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"]
Nov 04 00:18:33.709411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.709362    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"]
Nov 04 00:18:33.773029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.768935    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"]
Nov 04 00:18:33.899956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.899919    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"]
Nov 04 00:18:33.919525 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.919484    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"]
Nov 04 00:18:33.919945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:33.919874    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.031063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.031032    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:34.115641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115130    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.115641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115317    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllq7\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.115641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115464    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.115641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115638    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.116067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115692    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.116067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115769    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.116067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115821    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.116067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115865    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.116067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.115890    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.217573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.217538    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps\" (UniqueName: \"kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps\") pod \"55e17594-e4b1-4c74-9638-7dd2509a0e64\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") "
Nov 04 00:18:34.218063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.218036    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca\") pod \"55e17594-e4b1-4c74-9638-7dd2509a0e64\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") "
Nov 04 00:18:34.218335 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.218314    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca\") pod \"55e17594-e4b1-4c74-9638-7dd2509a0e64\" (UID: \"55e17594-e4b1-4c74-9638-7dd2509a0e64\") "
Nov 04 00:18:34.219344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.219315    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220062    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220230    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220376    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220458    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.218795    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca" pod "55e17594-e4b1-4c74-9638-7dd2509a0e64" (UID: "55e17594-e4b1-4c74-9638-7dd2509a0e64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:18:34.220637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220414    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps" pod "55e17594-e4b1-4c74-9638-7dd2509a0e64" (UID: "55e17594-e4b1-4c74-9638-7dd2509a0e64"). InnerVolumeSpecName "kube-api-access-nb5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:18:34.220693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220652    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220793    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.220921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.220905    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.221046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.221032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.221237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.221221    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllq7\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.221422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.221407    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.221703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.221685    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:34.221833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.221818    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb5ps\" (UniqueName: \"kubernetes.io/projected/55e17594-e4b1-4c74-9638-7dd2509a0e64-kube-api-access-nb5ps\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:34.222159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.222133    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.222311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.222289    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.222417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.221244    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.224759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.224713    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.226377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.226355    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.242160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.242123    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca" pod "55e17594-e4b1-4c74-9638-7dd2509a0e64" (UID: "55e17594-e4b1-4c74-9638-7dd2509a0e64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:18:34.242423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.242403    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllq7\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.243544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.243519    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.319851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.319814    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:34.323444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.323410    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e17594-e4b1-4c74-9638-7dd2509a0e64-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:34.666275 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:18:34.666225    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f45a4f9_10f2_43f3_bda6_492500f39719.slice/crio-c03e6be9a3c7b5cb849d2897f6c61ed2113f8622a495a9c2a6bbfc1348ef5751 WatchSource:0}: Error finding container c03e6be9a3c7b5cb849d2897f6c61ed2113f8622a495a9c2a6bbfc1348ef5751: Status 404 returned error can't find the container with id c03e6be9a3c7b5cb849d2897f6c61ed2113f8622a495a9c2a6bbfc1348ef5751
Nov 04 00:18:34.671662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.670832    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:34.671662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.670933    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:34.671662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.670977    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:34.681793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.681545    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"]
Nov 04 00:18:34.777465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.777407    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" event={"ID":"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520","Type":"ContainerStarted","Data":"f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab"}
Nov 04 00:18:34.777650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.777495    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" event={"ID":"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520","Type":"ContainerStarted","Data":"71520ebe61a3f5896542ee0e9248d9ad5ffc298a00f81cab2240123dd4a5e91a"}
Nov 04 00:18:34.780473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.780448    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx" event={"ID":"59b9616a-cc75-4b14-9af5-84c6bfd16cb7","Type":"ContainerStarted","Data":"685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26"}
Nov 04 00:18:34.780657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.780642    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx" event={"ID":"59b9616a-cc75-4b14-9af5-84c6bfd16cb7","Type":"ContainerStarted","Data":"993ba1028474f6fdc7ded96a90a2723bc65550d06a068e2b9af729f2794a06e5"}
Nov 04 00:18:34.781345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.781328    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:34.782782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.782757    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" event={"ID":"9f45a4f9-10f2-43f3-bda6-492500f39719","Type":"ContainerStarted","Data":"c03e6be9a3c7b5cb849d2897f6c61ed2113f8622a495a9c2a6bbfc1348ef5751"}
Nov 04 00:18:34.784520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.784495    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"
Nov 04 00:18:34.784612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.784545    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg" event={"ID":"55e17594-e4b1-4c74-9638-7dd2509a0e64","Type":"ContainerDied","Data":"f53e0c7b5a4a0778f557e8af24c9ff93b6b6c89bade8623654ea926d41be49a8"}
Nov 04 00:18:34.784612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.784589    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f53e0c7b5a4a0778f557e8af24c9ff93b6b6c89bade8623654ea926d41be49a8"
Nov 04 00:18:34.795834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.795810    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" event={"ID":"f7688723-0aea-4153-959e-4473873f281e","Type":"ContainerStarted","Data":"72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366"}
Nov 04 00:18:34.795980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.795961    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" event={"ID":"f7688723-0aea-4153-959e-4473873f281e","Type":"ContainerStarted","Data":"a19ea464cc25419cbafe7993e0fd73176fce54de93204e46802393ad4aee74fd"}
Nov 04 00:18:34.796076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.796010    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:34.834770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.833339    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podStartSLOduration=2.833320741 podStartE2EDuration="2.833320741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:34.820426393 +0000 UTC m=+2979.920026550" watchObservedRunningTime="2026-11-04 00:18:34.833320741 +0000 UTC m=+2979.932920885"
Nov 04 00:18:34.855615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.855558    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" podStartSLOduration=2.8555085829999998 podStartE2EDuration="2.855508583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:34.846627433 +0000 UTC m=+2979.946227577" watchObservedRunningTime="2026-11-04 00:18:34.855508583 +0000 UTC m=+2979.955108729"
Nov 04 00:18:34.868109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:34.868020    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx" podStartSLOduration=2.868005904 podStartE2EDuration="2.868005904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:34.867160388 +0000 UTC m=+2979.966760533" watchObservedRunningTime="2026-11-04 00:18:34.868005904 +0000 UTC m=+2979.967606047"
Nov 04 00:18:35.178818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:35.178782    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:35.180890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:35.180859    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Nov 04 00:18:35.181030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:35.180920    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Nov 04 00:18:35.823836 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:35.823793    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" event={"ID":"9f45a4f9-10f2-43f3-bda6-492500f39719","Type":"ContainerStarted","Data":"2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572"}
Nov 04 00:18:35.871643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:35.871561    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podStartSLOduration=2.871540091 podStartE2EDuration="2.871540091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:35.870105223 +0000 UTC m=+2980.969705359" watchObservedRunningTime="2026-11-04 00:18:35.871540091 +0000 UTC m=+2980.971140235"
Nov 04 00:18:36.179502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:36.179409    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Nov 04 00:18:36.179502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:36.179470    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Nov 04 00:18:36.322022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:36.321969    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:36.326897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:36.326864    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 04 00:18:36.327139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:36.327104    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 04 00:18:37.179860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:37.179816    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Nov 04 00:18:37.180274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:37.179874    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Nov 04 00:18:37.324288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:37.324242    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 04 00:18:37.324615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:37.324576    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 04 00:18:38.179573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:38.179531    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Nov 04 00:18:38.179773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:38.179587    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Nov 04 00:18:38.321764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:38.321137    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 04 00:18:38.321764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:38.321201    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 04 00:18:39.179113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:39.179079    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Nov 04 00:18:39.179335 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:39.179314    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Nov 04 00:18:39.332766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:39.327579    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 04 00:18:39.332766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:39.327643    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 04 00:18:40.179559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:40.179521    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Nov 04 00:18:40.179785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:40.179577    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Nov 04 00:18:40.324182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:40.324138    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 04 00:18:40.325763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:40.325711    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 04 00:18:41.185161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.185125    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:41.185774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.185588    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:41.186416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.186386    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:18:41.345888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.345849    5074 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 04 00:18:41.346219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.346143    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 04 00:18:41.422110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.422065    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"]
Nov 04 00:18:41.423613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.423581    5074 cpu_manager.go:414] "containerMap: removing container" podUID="55e17594-e4b1-4c74-9638-7dd2509a0e64" containerName="revoke-keys"
Nov 04 00:18:41.423613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.423610    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e17594-e4b1-4c74-9638-7dd2509a0e64" containerName="revoke-keys"
Nov 04 00:18:41.423977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.423931    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e17594-e4b1-4c74-9638-7dd2509a0e64" containerName="revoke-keys"
Nov 04 00:18:41.452454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.452364    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"]
Nov 04 00:18:41.452624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.452497    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.457061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.457038    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:18:41.457227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.457056    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:18:41.562063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.562019    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.562269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.562075    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ndn\" (UniqueName: \"kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.562347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.562321    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.664368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.664311    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.664368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.664372    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ndn\" (UniqueName: \"kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.664650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.664543    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.665124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.665097    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.667145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.667124    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.695188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.692713    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ndn\" (UniqueName: \"kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn\") pod \"maas-api-revoke-sub-f04d51c0-hn8qd\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:41.776980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:41.776894    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:42.101456 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:18:42.101421    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84fbd7a_913f_4f18_a539_d7ab7127c5b5.slice/crio-dd3a65cf435630424f8e8d70b46362868b7af4eb40e1315d7636850e7b0397a0 WatchSource:0}: Error finding container dd3a65cf435630424f8e8d70b46362868b7af4eb40e1315d7636850e7b0397a0: Status 404 returned error can't find the container with id dd3a65cf435630424f8e8d70b46362868b7af4eb40e1315d7636850e7b0397a0
Nov 04 00:18:42.101831 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:42.101804    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"]
Nov 04 00:18:42.327929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:42.327899    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:42.328553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:42.328499    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:42.329415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:42.329396    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:18:43.003630 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:43.003589    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd" event={"ID":"f84fbd7a-913f-4f18-a539-d7ab7127c5b5","Type":"ContainerStarted","Data":"4fb6179edd73c92bb498c93a58a95986445b235890fcb5d5344a5d37db693d06"}
Nov 04 00:18:43.003895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:43.003656    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd" event={"ID":"f84fbd7a-913f-4f18-a539-d7ab7127c5b5","Type":"ContainerStarted","Data":"dd3a65cf435630424f8e8d70b46362868b7af4eb40e1315d7636850e7b0397a0"}
Nov 04 00:18:43.025026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:43.024972    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd" podStartSLOduration=2.024959336 podStartE2EDuration="2.024959336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:43.022472209 +0000 UTC m=+2988.122072357" watchObservedRunningTime="2026-11-04 00:18:43.024959336 +0000 UTC m=+2988.124559493"
Nov 04 00:18:44.015963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:44.015932    5074 generic.go:356] "Generic (PLEG): container finished" podID="f84fbd7a-913f-4f18-a539-d7ab7127c5b5" containerID="4fb6179edd73c92bb498c93a58a95986445b235890fcb5d5344a5d37db693d06" exitCode=0
Nov 04 00:18:44.016449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:44.016037    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd" event={"ID":"f84fbd7a-913f-4f18-a539-d7ab7127c5b5","Type":"ContainerDied","Data":"4fb6179edd73c92bb498c93a58a95986445b235890fcb5d5344a5d37db693d06"}
Nov 04 00:18:45.233945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.233909    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:45.337964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.337859    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca\") pod \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") "
Nov 04 00:18:45.338145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.338103    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca\") pod \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") "
Nov 04 00:18:45.338213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.338156    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn\" (UniqueName: \"kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn\") pod \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\" (UID: \"f84fbd7a-913f-4f18-a539-d7ab7127c5b5\") "
Nov 04 00:18:45.338514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.338476    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca" pod "f84fbd7a-913f-4f18-a539-d7ab7127c5b5" (UID: "f84fbd7a-913f-4f18-a539-d7ab7127c5b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:18:45.338909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.338873    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:45.341985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.341954    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn" pod "f84fbd7a-913f-4f18-a539-d7ab7127c5b5" (UID: "f84fbd7a-913f-4f18-a539-d7ab7127c5b5"). InnerVolumeSpecName "kube-api-access-x9ndn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:18:45.342467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.342443    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca" pod "f84fbd7a-913f-4f18-a539-d7ab7127c5b5" (UID: "f84fbd7a-913f-4f18-a539-d7ab7127c5b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:18:45.441050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.441007    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:45.441050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.441045    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ndn\" (UniqueName: \"kubernetes.io/projected/f84fbd7a-913f-4f18-a539-d7ab7127c5b5-kube-api-access-x9ndn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:18:45.829621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:45.829589    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:18:46.043372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:46.043340    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"
Nov 04 00:18:46.043982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:46.043393    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd" event={"ID":"f84fbd7a-913f-4f18-a539-d7ab7127c5b5","Type":"ContainerDied","Data":"dd3a65cf435630424f8e8d70b46362868b7af4eb40e1315d7636850e7b0397a0"}
Nov 04 00:18:46.044141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:46.044008    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3a65cf435630424f8e8d70b46362868b7af4eb40e1315d7636850e7b0397a0"
Nov 04 00:18:46.874678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:46.874640    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:18:47.820123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.820082    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"]
Nov 04 00:18:47.822075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.822042    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f84fbd7a-913f-4f18-a539-d7ab7127c5b5" containerName="revoke-keys"
Nov 04 00:18:47.822273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.822242    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84fbd7a-913f-4f18-a539-d7ab7127c5b5" containerName="revoke-keys"
Nov 04 00:18:47.822672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.822650    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84fbd7a-913f-4f18-a539-d7ab7127c5b5" containerName="revoke-keys"
Nov 04 00:18:47.849142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.849106    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"]
Nov 04 00:18:47.849328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.849239    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.907878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.907841    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908388    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8lv\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908546    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908618    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908698    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908758    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908855    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.908954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908940    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:47.909124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:47.908968    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.019955    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020074    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020107    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020138    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020363    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8lv\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020431    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020449    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020608    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.020894    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.021675    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.021898    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.021990    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.022044    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.029312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.022468    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.042394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.041695    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.048307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.048213    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8lv\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.053659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.053625    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.058888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.058850    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.167228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.167146    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:48.483549 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:18:48.483512    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64c0883_ee43_46e9_baef_3a72361aced3.slice/crio-34b802ca8fcf333e8260f15f4db2e4e152ebf083bb6786cf12807597e012a4b3 WatchSource:0}: Error finding container 34b802ca8fcf333e8260f15f4db2e4e152ebf083bb6786cf12807597e012a4b3: Status 404 returned error can't find the container with id 34b802ca8fcf333e8260f15f4db2e4e152ebf083bb6786cf12807597e012a4b3
Nov 04 00:18:48.483943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.483916    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"]
Nov 04 00:18:48.487515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.487477    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:48.487622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.487561    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:48.487622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:48.487601    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:18:49.213760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:49.211500    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" event={"ID":"c64c0883-ee43-46e9-baef-3a72361aced3","Type":"ContainerStarted","Data":"3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e"}
Nov 04 00:18:49.213760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:49.211558    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" event={"ID":"c64c0883-ee43-46e9-baef-3a72361aced3","Type":"ContainerStarted","Data":"34b802ca8fcf333e8260f15f4db2e4e152ebf083bb6786cf12807597e012a4b3"}
Nov 04 00:18:49.243906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:49.243844    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podStartSLOduration=2.243830996 podStartE2EDuration="2.243830996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:49.236876689 +0000 UTC m=+2994.336476831" watchObservedRunningTime="2026-11-04 00:18:49.243830996 +0000 UTC m=+2994.343431137"
Nov 04 00:18:50.167926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:50.167888    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:50.170262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:50.169996    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 04 00:18:50.170262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:50.170083    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 04 00:18:51.173284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:51.168016    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 04 00:18:51.173284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:51.168089    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 04 00:18:52.168156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:52.168113    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 04 00:18:52.168314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:52.168175    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 04 00:18:53.167908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:53.167867    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 04 00:18:53.168330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:53.167933    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 04 00:18:54.167766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:54.167703    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 04 00:18:54.167946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:54.167783    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 04 00:18:55.173455 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:55.173422    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:55.174002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:55.173971    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:55.174918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:55.174887    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:18:55.827720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:55.827688    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:18:55.836371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:18:55.836337    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:18:58.032654 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:18:58.032614    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:19:04.248105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.248071    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"]
Nov 04 00:19:04.255369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.255336    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.261223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.261193    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"]
Nov 04 00:19:04.263579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.263555    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:19:04.263732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.263712    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:19:04.321672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.321632    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9fk\" (UniqueName: \"kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.322396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.322360    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.322637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.322593    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.424561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.424518    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.424803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.424652    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.424803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.424701    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9fk\" (UniqueName: \"kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.425306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.425281    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.427423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.427391    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.438308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.438272    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9fk\" (UniqueName: \"kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk\") pod \"maas-api-revoke-sub-bc3d0f51-lktjq\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.592620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.592544    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:04.889968 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:19:04.889935    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee03975_a1f9_4ebc_b64f_2dc45e16e1bf.slice/crio-a59bb85873f908cfd83a474365a260cbea06890272370eb08c8e9ffd853ff6ed WatchSource:0}: Error finding container a59bb85873f908cfd83a474365a260cbea06890272370eb08c8e9ffd853ff6ed: Status 404 returned error can't find the container with id a59bb85873f908cfd83a474365a260cbea06890272370eb08c8e9ffd853ff6ed
Nov 04 00:19:04.891719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:04.891570    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"]
Nov 04 00:19:05.456808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:05.456770    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq" event={"ID":"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf","Type":"ContainerStarted","Data":"abf4e68e0c4f400ca7cda6b333c4a6fc79339f1dfdd265430d4c0e03acd5f46b"}
Nov 04 00:19:05.457194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:05.456833    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq" event={"ID":"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf","Type":"ContainerStarted","Data":"a59bb85873f908cfd83a474365a260cbea06890272370eb08c8e9ffd853ff6ed"}
Nov 04 00:19:05.479790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:05.479716    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq" podStartSLOduration=1.479702789 podStartE2EDuration="1.479702789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:05.477412705 +0000 UTC m=+3010.577012849" watchObservedRunningTime="2026-11-04 00:19:05.479702789 +0000 UTC m=+3010.579302933"
Nov 04 00:19:06.468730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:06.468699    5074 generic.go:356] "Generic (PLEG): container finished" podID="9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" containerID="abf4e68e0c4f400ca7cda6b333c4a6fc79339f1dfdd265430d4c0e03acd5f46b" exitCode=0
Nov 04 00:19:06.469235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:06.468779    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq" event={"ID":"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf","Type":"ContainerDied","Data":"abf4e68e0c4f400ca7cda6b333c4a6fc79339f1dfdd265430d4c0e03acd5f46b"}
Nov 04 00:19:07.730305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.730235    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:07.740223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.740189    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca\") pod \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") "
Nov 04 00:19:07.740380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.740331    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk\" (UniqueName: \"kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk\") pod \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") "
Nov 04 00:19:07.740474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.740459    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca\") pod \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\" (UID: \"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf\") "
Nov 04 00:19:07.740860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.740823    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca" pod "9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" (UID: "9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:19:07.741269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.741245    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:07.742603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.742581    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk" pod "9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" (UID: "9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf"). InnerVolumeSpecName "kube-api-access-lx9fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:19:07.742853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.742831    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca" pod "9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" (UID: "9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:19:07.842588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.842547    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:07.842588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:07.842581    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9fk\" (UniqueName: \"kubernetes.io/projected/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf-kube-api-access-lx9fk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:08.488169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:08.488135    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"
Nov 04 00:19:08.488169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:08.488159    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq" event={"ID":"9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf","Type":"ContainerDied","Data":"a59bb85873f908cfd83a474365a260cbea06890272370eb08c8e9ffd853ff6ed"}
Nov 04 00:19:08.488397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:08.488208    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59bb85873f908cfd83a474365a260cbea06890272370eb08c8e9ffd853ff6ed"
Nov 04 00:19:14.310269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.310232    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"]
Nov 04 00:19:14.311174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.311156    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" containerName="revoke-keys"
Nov 04 00:19:14.311174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.311174    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" containerName="revoke-keys"
Nov 04 00:19:14.311375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.311364    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" containerName="revoke-keys"
Nov 04 00:19:14.315164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.315144    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.325894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.325865    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"]
Nov 04 00:19:14.328368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.328342    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-757a7d\""
Nov 04 00:19:14.328596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.328572    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-757a7d-dockercfg-mw6zx\""
Nov 04 00:19:14.387466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.387414    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"]
Nov 04 00:19:14.418714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.418668    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5w6b\" (UniqueName: \"kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b\") pod \"payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.419186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.419156    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.596817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.522993    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w6b\" (UniqueName: \"kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b\") pod \"payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.596817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.523178    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.596817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.524250    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.596817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.531953    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"]
Nov 04 00:19:14.596817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.532076    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.602330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.598129    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w6b\" (UniqueName: \"kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b\") pod \"payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.651669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.651282    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.651669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.651372    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7m4l\" (UniqueName: \"kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l\") pod \"payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.658138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.657928    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:14.754399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.753723    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.754399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.753828    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7m4l\" (UniqueName: \"kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l\") pod \"payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.754680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.754652    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.770092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.770061    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7m4l\" (UniqueName: \"kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l\") pod \"payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:14.933984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.933951    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"]
Nov 04 00:19:14.942265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.942233    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:14.946681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.946648    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"]
Nov 04 00:19:14.947182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.947160    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:19:14.952104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.949268    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:19:14.987098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:14.987065    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"]
Nov 04 00:19:14.988894 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:19:14.988861    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fc0539_0811_46b1_92f2_7d1062b6f0d5.slice/crio-1ab2bb0732d83efca92dc897feefb5330846e5ee54f17f1b029b914e3bef0859 WatchSource:0}: Error finding container 1ab2bb0732d83efca92dc897feefb5330846e5ee54f17f1b029b914e3bef0859: Status 404 returned error can't find the container with id 1ab2bb0732d83efca92dc897feefb5330846e5ee54f17f1b029b914e3bef0859
Nov 04 00:19:15.042112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.042082    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:15.059783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.059713    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.059957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.059935    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.060057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.059987    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p744b\" (UniqueName: \"kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.162119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.162085    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p744b\" (UniqueName: \"kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.162330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.162306    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.162463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.162447    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.163646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.163594    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.165014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.164988    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.176130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.176110    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p744b\" (UniqueName: \"kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b\") pod \"maas-api-revoke-sub-6b303246-gs5tr\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.264662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.264586    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:15.356250 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.356096    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"]
Nov 04 00:19:15.593548 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:19:15.593517    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0d7954_5862_4a8f_a614_5b1e215e0f07.slice/crio-b1587a91247abe7fdac4b1d2db047d3f30260e86430fe8321d0f4bef59a50ef0 WatchSource:0}: Error finding container b1587a91247abe7fdac4b1d2db047d3f30260e86430fe8321d0f4bef59a50ef0: Status 404 returned error can't find the container with id b1587a91247abe7fdac4b1d2db047d3f30260e86430fe8321d0f4bef59a50ef0
Nov 04 00:19:15.594507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.594479    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"]
Nov 04 00:19:15.635117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.635086    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr" event={"ID":"4e0d7954-5862-4a8f-a614-5b1e215e0f07","Type":"ContainerStarted","Data":"b1587a91247abe7fdac4b1d2db047d3f30260e86430fe8321d0f4bef59a50ef0"}
Nov 04 00:19:15.637172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.637141    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc" event={"ID":"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c","Type":"ContainerStarted","Data":"baecb5849a2ce461c6174dfb7e843870a2357c2414a9dd225fce4100471ce95c"}
Nov 04 00:19:15.637321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.637216    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:15.637321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.637241    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc" event={"ID":"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c","Type":"ContainerStarted","Data":"fce4a38dee2b37a9d35e41b3c0e5b99c2c1401a18e76edda701c2eacbbf74c1c"}
Nov 04 00:19:15.639214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.639182    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" event={"ID":"25fc0539-0811-46b1-92f2-7d1062b6f0d5","Type":"ContainerStarted","Data":"c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c"}
Nov 04 00:19:15.639336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.639238    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" event={"ID":"25fc0539-0811-46b1-92f2-7d1062b6f0d5","Type":"ContainerStarted","Data":"1ab2bb0732d83efca92dc897feefb5330846e5ee54f17f1b029b914e3bef0859"}
Nov 04 00:19:15.639336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.639266    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:15.725681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.725621    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc" podStartSLOduration=1.7256034260000002 podStartE2EDuration="1.725603426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:15.721780244 +0000 UTC m=+3020.821380390" watchObservedRunningTime="2026-11-04 00:19:15.725603426 +0000 UTC m=+3020.825203573"
Nov 04 00:19:15.752230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:15.752183    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" podStartSLOduration=1.752169684 podStartE2EDuration="1.752169684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:15.746193733 +0000 UTC m=+3020.845793879" watchObservedRunningTime="2026-11-04 00:19:15.752169684 +0000 UTC m=+3020.851769883"
Nov 04 00:19:16.551803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:16.551766    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"]
Nov 04 00:19:16.651981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:16.651943    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr" event={"ID":"4e0d7954-5862-4a8f-a614-5b1e215e0f07","Type":"ContainerStarted","Data":"70800aed2e453cc21f341064debea92d26a09bf083b8c5f317a4dbb08b530b86"}
Nov 04 00:19:16.676655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:16.676598    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr" podStartSLOduration=2.676583493 podStartE2EDuration="2.676583493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:16.673255282 +0000 UTC m=+3021.772855437" watchObservedRunningTime="2026-11-04 00:19:16.676583493 +0000 UTC m=+3021.776183637"
Nov 04 00:19:17.500111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.500071    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"]
Nov 04 00:19:17.662990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.662957    5074 generic.go:356] "Generic (PLEG): container finished" podID="4e0d7954-5862-4a8f-a614-5b1e215e0f07" containerID="70800aed2e453cc21f341064debea92d26a09bf083b8c5f317a4dbb08b530b86" exitCode=0
Nov 04 00:19:17.663407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.663041    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr" event={"ID":"4e0d7954-5862-4a8f-a614-5b1e215e0f07","Type":"ContainerDied","Data":"70800aed2e453cc21f341064debea92d26a09bf083b8c5f317a4dbb08b530b86"}
Nov 04 00:19:17.857824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.857701    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"]
Nov 04 00:19:17.863357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.863336    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.875320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.875288    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"]
Nov 04 00:19:17.917049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917006    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917137    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917215    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917295    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917458    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917531    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917579    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5k6\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917663    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:17.917793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:17.917763    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.019068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.019391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019153    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019667    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019753    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019868    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019959    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.019994    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5k6\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.020043    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.020134    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.020499    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.020674    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.020938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.020916    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.022456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.021381    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.022456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.021698    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.022641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.022549    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.043381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.043309    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.045700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.045668    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.051603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.051575    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5k6\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.181950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.181868    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:18.484429 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:19:18.484398    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9bec50_4989_4a9f_a4ce_46cc01ef5b04.slice/crio-df7a4c78b8153a766da9f203829fa4b6385b8ebaca0be0c002b8f2db255d803b WatchSource:0}: Error finding container df7a4c78b8153a766da9f203829fa4b6385b8ebaca0be0c002b8f2db255d803b: Status 404 returned error can't find the container with id df7a4c78b8153a766da9f203829fa4b6385b8ebaca0be0c002b8f2db255d803b
Nov 04 00:19:18.485206 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.485177    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"]
Nov 04 00:19:18.487876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.487840    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:19:18.487974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.487920    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:19:18.487974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.487952    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:19:18.677417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.677379    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" event={"ID":"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04","Type":"ContainerStarted","Data":"e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8"}
Nov 04 00:19:18.677919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.677449    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" event={"ID":"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04","Type":"ContainerStarted","Data":"df7a4c78b8153a766da9f203829fa4b6385b8ebaca0be0c002b8f2db255d803b"}
Nov 04 00:19:18.727100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.725904    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podStartSLOduration=1.725885362 podStartE2EDuration="1.725885362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:18.703779682 +0000 UTC m=+3023.803379824" watchObservedRunningTime="2026-11-04 00:19:18.725885362 +0000 UTC m=+3023.825485506"
Nov 04 00:19:18.905264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.905230    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:18.941890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.938638    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca\") pod \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") "
Nov 04 00:19:18.941890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.938832    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b\" (UniqueName: \"kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b\") pod \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") "
Nov 04 00:19:18.941890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.938891    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca\") pod \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\" (UID: \"4e0d7954-5862-4a8f-a614-5b1e215e0f07\") "
Nov 04 00:19:18.953331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.949532    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b" pod "4e0d7954-5862-4a8f-a614-5b1e215e0f07" (UID: "4e0d7954-5862-4a8f-a614-5b1e215e0f07"). InnerVolumeSpecName "kube-api-access-p744b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:19:18.959523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.959491    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca" pod "4e0d7954-5862-4a8f-a614-5b1e215e0f07" (UID: "4e0d7954-5862-4a8f-a614-5b1e215e0f07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:19:18.959684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:18.959669    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca" pod "4e0d7954-5862-4a8f-a614-5b1e215e0f07" (UID: "4e0d7954-5862-4a8f-a614-5b1e215e0f07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:19:19.048041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.043397    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:19.048041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.043512    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0d7954-5862-4a8f-a614-5b1e215e0f07-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:19.048041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.043532    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p744b\" (UniqueName: \"kubernetes.io/projected/4e0d7954-5862-4a8f-a614-5b1e215e0f07-kube-api-access-p744b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:19.183007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.182968    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:19.184832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.184799    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 04 00:19:19.184973 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.184863    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 04 00:19:19.688529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.688501    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"
Nov 04 00:19:19.688948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.688538    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr" event={"ID":"4e0d7954-5862-4a8f-a614-5b1e215e0f07","Type":"ContainerDied","Data":"b1587a91247abe7fdac4b1d2db047d3f30260e86430fe8321d0f4bef59a50ef0"}
Nov 04 00:19:19.688948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:19.688587    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1587a91247abe7fdac4b1d2db047d3f30260e86430fe8321d0f4bef59a50ef0"
Nov 04 00:19:20.182510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:20.182424    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 04 00:19:20.182510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:20.182482    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 04 00:19:21.182600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:21.182562    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 04 00:19:21.183039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:21.182630    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 04 00:19:21.552073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:21.552029    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4" podUID="b4fe9b48-b66d-46ce-8f3e-33472837a9a6" containerName="payload-processing" containerID="cri-o://3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3" gracePeriod=25
Nov 04 00:19:21.751988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:21.751942    5074 generic.go:356] "Generic (PLEG): container finished" podID="b4fe9b48-b66d-46ce-8f3e-33472837a9a6" containerID="3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3" exitCode=0
Nov 04 00:19:21.752168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:21.752096    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4" event={"ID":"b4fe9b48-b66d-46ce-8f3e-33472837a9a6","Type":"ContainerDied","Data":"3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3"}
Nov 04 00:19:21.764852 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:19:21.763329    5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fe9b48_b66d_46ce_8f3e_33472837a9a6.slice/crio-3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fe9b48_b66d_46ce_8f3e_33472837a9a6.slice/crio-conmon-3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:19:21.988387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:21.988356    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:19:22.154495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.152311    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials\") pod \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") "
Nov 04 00:19:22.154495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.152355    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume\") pod \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") "
Nov 04 00:19:22.154495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.152378    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg\") pod \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") "
Nov 04 00:19:22.154495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.152411    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume\") pod \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\" (UID: \"b4fe9b48-b66d-46ce-8f3e-33472837a9a6\") "
Nov 04 00:19:22.154495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.153793    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume" pod "b4fe9b48-b66d-46ce-8f3e-33472837a9a6" (UID: "b4fe9b48-b66d-46ce-8f3e-33472837a9a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:19:22.155373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.155345    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume" pod "b4fe9b48-b66d-46ce-8f3e-33472837a9a6" (UID: "b4fe9b48-b66d-46ce-8f3e-33472837a9a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:19:22.155574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.155545    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg" pod "b4fe9b48-b66d-46ce-8f3e-33472837a9a6" (UID: "b4fe9b48-b66d-46ce-8f3e-33472837a9a6"). InnerVolumeSpecName "kube-api-access-8c6mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:19:22.156672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.156648    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials" pod "b4fe9b48-b66d-46ce-8f3e-33472837a9a6" (UID: "b4fe9b48-b66d-46ce-8f3e-33472837a9a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:19:22.185162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.183071    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 04 00:19:22.185162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.183131    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 04 00:19:22.254721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.254638    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:22.254721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.254674    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:22.254721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.254689    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:22.254721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.254704    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6mg\" (UniqueName: \"kubernetes.io/projected/b4fe9b48-b66d-46ce-8f3e-33472837a9a6-kube-api-access-8c6mg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:22.501875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.501830    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m" podUID="9a070641-54bd-469c-87f2-43b98182c7fe" containerName="payload-processing" containerID="cri-o://205c11cedc983231b2b0701402fe0d771fecdf8d46765337d9aad0788562ab1c" gracePeriod=25
Nov 04 00:19:22.775034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.774406    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4" event={"ID":"b4fe9b48-b66d-46ce-8f3e-33472837a9a6","Type":"ContainerDied","Data":"8c82765bbfdde416c3c29f05e256c4a019fda6991277590771e2f78500e9ea47"}
Nov 04 00:19:22.775034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.774480    5074 scope.go:122] "RemoveContainer" containerID="3760ffdf09b8e1d892f969eea0656ca5b54bfcb37f280b5d763ae04e4d6b37b3"
Nov 04 00:19:22.775034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.774639    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"
Nov 04 00:19:22.778804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.778491    5074 generic.go:356] "Generic (PLEG): container finished" podID="9a070641-54bd-469c-87f2-43b98182c7fe" containerID="205c11cedc983231b2b0701402fe0d771fecdf8d46765337d9aad0788562ab1c" exitCode=0
Nov 04 00:19:22.778804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.778542    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m" event={"ID":"9a070641-54bd-469c-87f2-43b98182c7fe","Type":"ContainerDied","Data":"205c11cedc983231b2b0701402fe0d771fecdf8d46765337d9aad0788562ab1c"}
Nov 04 00:19:22.845805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.845771    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"]
Nov 04 00:19:22.854787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.854319    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-8558lg9f4"]
Nov 04 00:19:22.932894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:22.932863    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:19:23.052285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.052186    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fe9b48-b66d-46ce-8f3e-33472837a9a6" path="/var/lib/kubelet/pods/b4fe9b48-b66d-46ce-8f3e-33472837a9a6/volumes"
Nov 04 00:19:23.076075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.076033    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume\") pod \"9a070641-54bd-469c-87f2-43b98182c7fe\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") "
Nov 04 00:19:23.076364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.076343    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials\") pod \"9a070641-54bd-469c-87f2-43b98182c7fe\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") "
Nov 04 00:19:23.076437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.076410    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj\") pod \"9a070641-54bd-469c-87f2-43b98182c7fe\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") "
Nov 04 00:19:23.076437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.076431    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume\") pod \"9a070641-54bd-469c-87f2-43b98182c7fe\" (UID: \"9a070641-54bd-469c-87f2-43b98182c7fe\") "
Nov 04 00:19:23.077842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.077800    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume" pod "9a070641-54bd-469c-87f2-43b98182c7fe" (UID: "9a070641-54bd-469c-87f2-43b98182c7fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:19:23.077985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.077917    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume" pod "9a070641-54bd-469c-87f2-43b98182c7fe" (UID: "9a070641-54bd-469c-87f2-43b98182c7fe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:19:23.079753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.079723    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj" pod "9a070641-54bd-469c-87f2-43b98182c7fe" (UID: "9a070641-54bd-469c-87f2-43b98182c7fe"). InnerVolumeSpecName "kube-api-access-jq5cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:19:23.080119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.080096    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials" pod "9a070641-54bd-469c-87f2-43b98182c7fe" (UID: "9a070641-54bd-469c-87f2-43b98182c7fe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:19:23.179150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.179104    5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-provider-credentials\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:23.179150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.179142    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5cj\" (UniqueName: \"kubernetes.io/projected/9a070641-54bd-469c-87f2-43b98182c7fe-kube-api-access-jq5cj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:23.179150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.179156    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:23.179481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.179167    5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a070641-54bd-469c-87f2-43b98182c7fe-routing-overlay-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:19:23.182704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.182668    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 04 00:19:23.182881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.182734    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 04 00:19:23.790615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.790577    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"
Nov 04 00:19:23.790615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.790608    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m" event={"ID":"9a070641-54bd-469c-87f2-43b98182c7fe","Type":"ContainerDied","Data":"2b501c0b6cb672d9ed9a3ab62bdf684d62bcc4ed6989cb15606141ff7809f5b8"}
Nov 04 00:19:23.791194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.790675    5074 scope.go:122] "RemoveContainer" containerID="205c11cedc983231b2b0701402fe0d771fecdf8d46765337d9aad0788562ab1c"
Nov 04 00:19:23.839253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.839210    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"]
Nov 04 00:19:23.844919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:23.844892    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-7b88vrw4m"]
Nov 04 00:19:24.182598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:24.182508    5074 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 04 00:19:24.182598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:24.182565    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 04 00:19:25.058208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:25.058167    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a070641-54bd-469c-87f2-43b98182c7fe" path="/var/lib/kubelet/pods/9a070641-54bd-469c-87f2-43b98182c7fe/volumes"
Nov 04 00:19:25.295684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:25.295648    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:25.296490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:25.296464    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:25.297842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:25.297817    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:19:26.681070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:26.681038    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:19:26.682914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:26.682891    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:19:28.028061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:19:28.028022    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:19:39.790896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.790863    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"]
Nov 04 00:19:39.791855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.791836    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b4fe9b48-b66d-46ce-8f3e-33472837a9a6" containerName="payload-processing"
Nov 04 00:19:39.791915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.791856    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fe9b48-b66d-46ce-8f3e-33472837a9a6" containerName="payload-processing"
Nov 04 00:19:39.791953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.791919    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9a070641-54bd-469c-87f2-43b98182c7fe" containerName="payload-processing"
Nov 04 00:19:39.791953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.791925    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a070641-54bd-469c-87f2-43b98182c7fe" containerName="payload-processing"
Nov 04 00:19:39.791953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.791943    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4e0d7954-5862-4a8f-a614-5b1e215e0f07" containerName="revoke-keys"
Nov 04 00:19:39.791953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.791949    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0d7954-5862-4a8f-a614-5b1e215e0f07" containerName="revoke-keys"
Nov 04 00:19:39.792097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.792088    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a070641-54bd-469c-87f2-43b98182c7fe" containerName="payload-processing"
Nov 04 00:19:39.792135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.792101    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0d7954-5862-4a8f-a614-5b1e215e0f07" containerName="revoke-keys"
Nov 04 00:19:39.792135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.792113    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fe9b48-b66d-46ce-8f3e-33472837a9a6" containerName="payload-processing"
Nov 04 00:19:39.796461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.796436    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:39.802806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.802779    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"]
Nov 04 00:19:39.805391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.805367    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ecd46b\""
Nov 04 00:19:39.805391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.805385    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ecd46b-dockercfg-862rs\""
Nov 04 00:19:39.849780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.840970    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"]
Nov 04 00:19:39.850502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.850204    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"]
Nov 04 00:19:39.850502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.850315    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:39.912794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.912608    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:39.913008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:39.912782    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzljg\" (UniqueName: \"kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg\") pod \"payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:40.112373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.110092    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.112373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.110159    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qh76\" (UniqueName: \"kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76\") pod \"payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.113319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.110277    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:40.113319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.112827    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:40.113319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.112866    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzljg\" (UniqueName: \"kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg\") pod \"payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:40.131701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.131655    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzljg\" (UniqueName: \"kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg\") pod \"payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:40.135980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.135913    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:40.217982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.217947    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.218131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.218027    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qh76\" (UniqueName: \"kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76\") pod \"payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.219829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.219792    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.239252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.239223    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qh76\" (UniqueName: \"kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76\") pod \"payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.248521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.247513    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:40.449333 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:19:40.448573    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12115daa_a907_406d_ba0d_1b8b664d24ef.slice/crio-6b25fedf7ca759cbc111abdc3246317680588f5e126f637229a0f92279640963 WatchSource:0}: Error finding container 6b25fedf7ca759cbc111abdc3246317680588f5e126f637229a0f92279640963: Status 404 returned error can't find the container with id 6b25fedf7ca759cbc111abdc3246317680588f5e126f637229a0f92279640963
Nov 04 00:19:40.452929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.451768    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"]
Nov 04 00:19:40.579465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:40.579438    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"]
Nov 04 00:19:40.579605 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:19:40.579583    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8b2a7f_a0b7_43d4_b917_20f3d4f0062d.slice/crio-7c2aca4307487923a8e3cd2992e30d5c1eb06654aa2079e9aaa88cc1fcc5de99 WatchSource:0}: Error finding container 7c2aca4307487923a8e3cd2992e30d5c1eb06654aa2079e9aaa88cc1fcc5de99: Status 404 returned error can't find the container with id 7c2aca4307487923a8e3cd2992e30d5c1eb06654aa2079e9aaa88cc1fcc5de99
Nov 04 00:19:41.137036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.136988    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" event={"ID":"12115daa-a907-406d-ba0d-1b8b664d24ef","Type":"ContainerStarted","Data":"006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979"}
Nov 04 00:19:41.137546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.137057    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" event={"ID":"12115daa-a907-406d-ba0d-1b8b664d24ef","Type":"ContainerStarted","Data":"6b25fedf7ca759cbc111abdc3246317680588f5e126f637229a0f92279640963"}
Nov 04 00:19:41.137546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.137518    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:41.140198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.140168    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" event={"ID":"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d","Type":"ContainerStarted","Data":"c026b764909785e24672d906fce87071c884e8ee2aeab52b32c5faeaef166845"}
Nov 04 00:19:41.140356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.140217    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" event={"ID":"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d","Type":"ContainerStarted","Data":"7c2aca4307487923a8e3cd2992e30d5c1eb06654aa2079e9aaa88cc1fcc5de99"}
Nov 04 00:19:41.140356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.140243    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:41.163582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.163537    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" podStartSLOduration=2.16352409 podStartE2EDuration="2.16352409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:41.156953466 +0000 UTC m=+3046.256553610" watchObservedRunningTime="2026-11-04 00:19:41.16352409 +0000 UTC m=+3046.263124279"
Nov 04 00:19:41.182256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:41.182202    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" podStartSLOduration=2.1821875569999998 podStartE2EDuration="2.182187557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:41.176560718 +0000 UTC m=+3046.276160864" watchObservedRunningTime="2026-11-04 00:19:41.182187557 +0000 UTC m=+3046.281787700"
Nov 04 00:19:52.169493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:52.168541    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:19:52.179593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:52.179564    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:19:58.033188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:58.033151    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"]
Nov 04 00:19:58.033920 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:19:58.033890    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:19:58.042424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:58.042396    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a120cec-pclcq"]
Nov 04 00:19:59.053435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:19:59.053392    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62091dd9-fd32-4fa1-afd5-7ba971f2a163" path="/var/lib/kubelet/pods/62091dd9-fd32-4fa1-afd5-7ba971f2a163/volumes"
Nov 04 00:20:00.249487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:00.249452    5074 scope.go:122] "RemoveContainer" containerID="609e0d2816c98d761a68013dc6a6b6293ac5dec2f8abc1a256ede6e017ffb779"
Nov 04 00:20:28.033770 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:20:28.033164    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:20:32.429714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:32.429665    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" podUID="14f459d1824c1ea630aa90a99306c647" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:20:33.033019 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:33.032978    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"]
Nov 04 00:20:33.037925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:33.037894    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f677d1a6-5qqhw"]
Nov 04 00:20:35.023195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:35.023159    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2e61f6-765a-454d-a78b-02ef55d71942" path="/var/lib/kubelet/pods/4e2e61f6-765a-454d-a78b-02ef55d71942/volumes"
Nov 04 00:20:37.037982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:37.037926    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"]
Nov 04 00:20:37.044654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:37.044597    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4820179-nmldl"]
Nov 04 00:20:39.022238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:39.022198    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49061992-7aea-4bd6-a7e8-5ba4b5f4cf25" path="/var/lib/kubelet/pods/49061992-7aea-4bd6-a7e8-5ba4b5f4cf25/volumes"
Nov 04 00:20:45.033529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:45.033486    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"]
Nov 04 00:20:45.038402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:45.038373    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13c6a8e-qfvfn"]
Nov 04 00:20:45.548193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:45.548157    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"]
Nov 04 00:20:45.946233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:45.946200    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:45.970147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:45.970117    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6382fc-openshift-default-dockercfg-lb6jw\""
Nov 04 00:20:45.987775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:45.987093    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"]
Nov 04 00:20:46.069595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.069548    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.069931    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070069    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56t78\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070354    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070509    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070560    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070616    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.070807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070783    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.071000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.070984    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.172594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172554    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.172838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172634    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.172838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172678    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.172838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172708    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.172838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172735    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.172838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172794    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.172893    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173031    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173132    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56t78\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173163    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173428    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173681    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.173910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.173758    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.175505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.175483    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.175722 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.175698    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.190183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.190116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56t78\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.203038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.202958    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token\") pod \"e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") " pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.295142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.295109    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:46.597397 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:20:46.597362    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113595e4_c554_4735_8d85_6e719e9b8c24.slice/crio-0e43cf7fe38d1f751e404f8f09d7ac9710e57dfbcb74937319d81c291217a1bf WatchSource:0}: Error finding container 0e43cf7fe38d1f751e404f8f09d7ac9710e57dfbcb74937319d81c291217a1bf: Status 404 returned error can't find the container with id 0e43cf7fe38d1f751e404f8f09d7ac9710e57dfbcb74937319d81c291217a1bf
Nov 04 00:20:46.604829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.601081    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:20:46.604829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.601173    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:20:46.604829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.601215    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:20:46.604829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:46.601448    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"]
Nov 04 00:20:47.062951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.062904    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282d83c3-dd16-4fee-aefe-63e45f2f51f5" path="/var/lib/kubelet/pods/282d83c3-dd16-4fee-aefe-63e45f2f51f5/volumes"
Nov 04 00:20:47.063781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.063450    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" event={"ID":"113595e4-c554-4735-8d85-6e719e9b8c24","Type":"ContainerStarted","Data":"b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"}
Nov 04 00:20:47.063781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.063519    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"]
Nov 04 00:20:47.063781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.063544    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" event={"ID":"113595e4-c554-4735-8d85-6e719e9b8c24","Type":"ContainerStarted","Data":"0e43cf7fe38d1f751e404f8f09d7ac9710e57dfbcb74937319d81c291217a1bf"}
Nov 04 00:20:47.066422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.066387    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-144b2efc-rbdvn"]
Nov 04 00:20:47.091287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.091234    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podStartSLOduration=2.091221369 podStartE2EDuration="2.091221369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:20:47.088700892 +0000 UTC m=+3112.188301038" watchObservedRunningTime="2026-11-04 00:20:47.091221369 +0000 UTC m=+3112.190821515"
Nov 04 00:20:47.295815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.295780    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:47.307998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:47.307964    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:48.068884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:48.068851    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:48.071096 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:48.069858    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:20:49.023629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:49.023589    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c123ea33-cf7c-41eb-9eca-70fd8119c4ec" path="/var/lib/kubelet/pods/c123ea33-cf7c-41eb-9eca-70fd8119c4ec/volumes"
Nov 04 00:20:50.585276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.585244    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"]
Nov 04 00:20:50.594331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.591475    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.617903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.617869    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6382fc\""
Nov 04 00:20:50.629923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.629891    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"]
Nov 04 00:20:50.683007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.682965    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.683431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.683398    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snws\" (UniqueName: \"kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws\") pod \"payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.817768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.816902    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.817768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.817109    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snws\" (UniqueName: \"kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws\") pod \"payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.818591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.818558    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.872815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.872103    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snws\" (UniqueName: \"kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws\") pod \"payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:50.968763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:50.966049    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:51.331478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:51.331431    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"]
Nov 04 00:20:51.331681 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:20:51.331503    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da8a195_2855_42e0_8035_e8991fd5b0fd.slice/crio-669742f147b06feddc01954b91cfb70acacc612fd49fb831c1c82899be336121 WatchSource:0}: Error finding container 669742f147b06feddc01954b91cfb70acacc612fd49fb831c1c82899be336121: Status 404 returned error can't find the container with id 669742f147b06feddc01954b91cfb70acacc612fd49fb831c1c82899be336121
Nov 04 00:20:52.109933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:52.109896    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5" event={"ID":"3da8a195-2855-42e0-8035-e8991fd5b0fd","Type":"ContainerStarted","Data":"f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df"}
Nov 04 00:20:52.110450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:52.109956    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5" event={"ID":"3da8a195-2855-42e0-8035-e8991fd5b0fd","Type":"ContainerStarted","Data":"669742f147b06feddc01954b91cfb70acacc612fd49fb831c1c82899be336121"}
Nov 04 00:20:52.110450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:52.109983    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:20:52.132014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:20:52.131949    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5" podStartSLOduration=2.131931046 podStartE2EDuration="2.131931046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:20:52.130144819 +0000 UTC m=+3117.229744963" watchObservedRunningTime="2026-11-04 00:20:52.131931046 +0000 UTC m=+3117.231531190"
Nov 04 00:20:58.145766 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:20:58.139423    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:21:00.296233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:00.296198    5074 scope.go:122] "RemoveContainer" containerID="bfe7821f8a64b422aa1014557ad9df99e5e1011e5fc1ca04bf06c0475ab502fd"
Nov 04 00:21:03.124804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:03.124771    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:21:17.514873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.514841    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"]
Nov 04 00:21:17.543422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.535918    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.543422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.540598    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"]
Nov 04 00:21:17.544860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.544836    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6382fc-serving-cert\""
Nov 04 00:21:17.547828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.547804    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\""
Nov 04 00:21:17.547959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.547893    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 04 00:21:17.652046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.651285    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.652046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.651421    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggn2b\" (UniqueName: \"kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.652046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.651460    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.652046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.651574    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.754548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.754511    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.754548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.754567    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggn2b\" (UniqueName: \"kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.754957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.754600    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.754957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.754703    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.755893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.755863    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.757970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.757940    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.758263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.758244    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.768034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.767980    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggn2b\" (UniqueName: \"kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b\") pod \"maas-api-e2e-inf-b-6382fc-559588688-hjt6k\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:17.861090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:17.861054    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:18.340760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:18.340304    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:21:18.369779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:18.361201    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"]
Nov 04 00:21:18.440622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:18.440581    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" event={"ID":"50149425-2487-4c92-b0e1-9790ef804e3e","Type":"ContainerStarted","Data":"6bffd4f64ae8dea3650e7f5b3d12ae3c3b4a19ad3f854aa6ba4af751c78b5465"}
Nov 04 00:21:20.463423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:20.463334    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" event={"ID":"50149425-2487-4c92-b0e1-9790ef804e3e","Type":"ContainerStarted","Data":"f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"}
Nov 04 00:21:20.463423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:20.463408    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:20.488983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:20.488930    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" podStartSLOduration=1.617160602 podStartE2EDuration="3.488917294s" totalImagesPullingTime="1.871756692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:21:20.483694806 +0000 UTC m=+3145.583294950" watchObservedRunningTime="2026-11-04 00:21:20.488917294 +0000 UTC m=+3145.588517492"
Nov 04 00:21:26.480880 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:26.480824    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.123:8443/health\": dial tcp 10.132.1.123:8443: connect: connection refused"
Nov 04 00:21:27.244109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.244073    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"]
Nov 04 00:21:27.249336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.249313    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.256111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.256086    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d94e94-openshift-default-dockercfg-dfzz6\""
Nov 04 00:21:27.261545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.261008    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"]
Nov 04 00:21:27.300237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300200    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300372    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300403    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300472    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxh7\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300503    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300583    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300696    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300795    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.300855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.300840    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.402593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.402540    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.402889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.402644    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.402889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.402725    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.402941    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.402986    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.403037    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxh7\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.403066    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.403178    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.403245    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.403794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.403763    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.404095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.404069    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.404220 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.404191    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.404389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.404365    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.404498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.404442    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.407242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.407218    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.407557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.407530    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.417128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.417106    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.417225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.417110    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxh7\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.570452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.570356    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:27.889407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.889358    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"]
Nov 04 00:21:27.893607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.893571    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:21:27.893781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.893653    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:21:27.893781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:27.893696    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:21:28.054432 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:21:28.054387    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:21:28.557270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:28.557223    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" event={"ID":"430a67a5-71d7-4766-ad0f-bf046a861e53","Type":"ContainerStarted","Data":"21ae62c9a7f6d97adedcab97a82b7efa21d742b1377ba9273f4f50d72f3ca862"}
Nov 04 00:21:28.557472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:28.557334    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" event={"ID":"430a67a5-71d7-4766-ad0f-bf046a861e53","Type":"ContainerStarted","Data":"6786ffc77eae27bb2c73f59d14746b122777a43f2369719bdf7733af5c8966f9"}
Nov 04 00:21:28.570929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:28.570877    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:28.586473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:28.586403    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podStartSLOduration=1.586384419 podStartE2EDuration="1.586384419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:21:28.583065578 +0000 UTC m=+3153.682665722" watchObservedRunningTime="2026-11-04 00:21:28.586384419 +0000 UTC m=+3153.685984564"
Nov 04 00:21:29.572518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:29.572478    5074 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body=
Nov 04 00:21:29.572936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:29.572532    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused"
Nov 04 00:21:30.570753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:30.570707    5074 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body=
Nov 04 00:21:30.570940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:30.570777    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused"
Nov 04 00:21:30.578113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:30.578085    5074 generic.go:356] "Generic (PLEG): container finished" podID="50149425-2487-4c92-b0e1-9790ef804e3e" containerID="f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a" exitCode=1
Nov 04 00:21:30.578472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:30.578120    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" event={"ID":"50149425-2487-4c92-b0e1-9790ef804e3e","Type":"ContainerDied","Data":"f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"}
Nov 04 00:21:30.578599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:30.578579    5074 scope.go:122] "RemoveContainer" containerID="f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"
Nov 04 00:21:31.570771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:31.570720    5074 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body=
Nov 04 00:21:31.571011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:31.570797    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused"
Nov 04 00:21:31.590160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:31.590120    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" event={"ID":"50149425-2487-4c92-b0e1-9790ef804e3e","Type":"ContainerStarted","Data":"806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af"}
Nov 04 00:21:31.590584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:31.590346    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:32.577197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:32.577153    5074 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body=
Nov 04 00:21:32.577588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:32.577549    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused"
Nov 04 00:21:33.571725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:33.571683    5074 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body=
Nov 04 00:21:33.572169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:33.571771    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused"
Nov 04 00:21:34.680109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:34.680073    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:34.680617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:34.680572    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:34.681556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:34.681531    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:21:37.644842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:37.644808    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:21:42.258400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:42.258343    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"]
Nov 04 00:21:42.258971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:42.258607    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" containerID="cri-o://5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d" gracePeriod=30
Nov 04 00:21:42.269238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:42.269200    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"]
Nov 04 00:21:42.269883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:42.269827    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" containerID="cri-o://6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57" gracePeriod=30
Nov 04 00:21:42.277940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:42.277913    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"]
Nov 04 00:21:42.278165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:42.278137    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy" containerID="cri-o://4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273" gracePeriod=30
Nov 04 00:21:46.688538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:46.688498    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ln7lf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:21:46.689025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:46.688575    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:21:46.928007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:46.927976    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8s5gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:21:46.928186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:46.928038    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:21:47.753970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.753941    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:21:47.774723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.774691    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:21:47.783488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.783458    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:21:47.841989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.841907    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842043    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842097    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842129    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842167    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842181    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842217    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842265    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842310    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842329    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842346    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842389    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842484    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842510    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842534    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842569    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842614    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842660    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842729    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842849    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.842951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.842901    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843015    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843065    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843102    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert\") pod \"324fe1c5-df49-4e36-92d0-f4dccead361a\" (UID: \"324fe1c5-df49-4e36-92d0-f4dccead361a\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843135    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843164    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843214    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843263    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token\") pod \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\" (UID: \"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7\") "
Nov 04 00:21:47.844061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.843313    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket\") pod \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\" (UID: \"d16665b8-d193-41b7-a0b4-32108cbf6a6f\") "
Nov 04 00:21:47.844485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.844318    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.844485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.844344    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.844664 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.844602    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.846444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.846418    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.846794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.846773    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:21:47.847248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.847010    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.847248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.847189    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.848320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.847854    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:21:47.852243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.852209    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:21:47.852543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.852514    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.852868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.852844    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:21:47.854808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.854721    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.855046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.855014    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.855369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.855344    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.855681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.855658    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.857574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.857545    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.857757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.857717    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "kube-api-access-sxz6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.857859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.848077    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.860305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.860280    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.870384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.870362    5074 generic.go:356] "Generic (PLEG): container finished" podID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerID="4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273" exitCode=0
Nov 04 00:21:47.870600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.870584    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" event={"ID":"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7","Type":"ContainerDied","Data":"4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273"}
Nov 04 00:21:47.870765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.870731    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq" event={"ID":"4f32ec71-4f6a-49c4-9ba5-f6c931e159d7","Type":"ContainerDied","Data":"47d86b2198484faee6af7123a4fa5c90df36393652eadf80295e4014a7776eaf"}
Nov 04 00:21:47.870854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.870839    5074 scope.go:122] "RemoveContainer" containerID="4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273"
Nov 04 00:21:47.871055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.871037    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"
Nov 04 00:21:47.873226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.872298    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.873411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.873366    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.873872    5074 generic.go:356] "Generic (PLEG): container finished" podID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerID="5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d" exitCode=0
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.873914    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.874059    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" event={"ID":"324fe1c5-df49-4e36-92d0-f4dccead361a","Type":"ContainerDied","Data":"5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d"}
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.874117    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf" event={"ID":"324fe1c5-df49-4e36-92d0-f4dccead361a","Type":"ContainerDied","Data":"3b385eb75bab47960952c254e43a24447d68392580a7869b867a0c6306ad55a7"}
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.874221    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.874814    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs" pod "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" (UID: "4f32ec71-4f6a-49c4-9ba5-f6c931e159d7"). InnerVolumeSpecName "kube-api-access-88bqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.875214    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "kube-api-access-7h6lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.877766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.875494    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:21:47.878845 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.878765    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy" pod "324fe1c5-df49-4e36-92d0-f4dccead361a" (UID: "324fe1c5-df49-4e36-92d0-f4dccead361a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.885299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.882180    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token" pod "d16665b8-d193-41b7-a0b4-32108cbf6a6f" (UID: "d16665b8-d193-41b7-a0b4-32108cbf6a6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.885299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.882443    5074 generic.go:356] "Generic (PLEG): container finished" podID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerID="6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57" exitCode=0
Nov 04 00:21:47.885299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.882520    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" event={"ID":"d16665b8-d193-41b7-a0b4-32108cbf6a6f","Type":"ContainerDied","Data":"6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57"}
Nov 04 00:21:47.885299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.882567    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs" event={"ID":"d16665b8-d193-41b7-a0b4-32108cbf6a6f","Type":"ContainerDied","Data":"4ca6af0a79a511e394da5c4780ab3e04ca0d2908d218f79cef392db3d0a0d70c"}
Nov 04 00:21:47.885299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.882696    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"
Nov 04 00:21:47.897897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.897828    5074 scope.go:122] "RemoveContainer" containerID="4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273"
Nov 04 00:21:47.898181 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:21:47.898158    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273\": container with ID starting with 4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273 not found: ID does not exist" containerID="4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273"
Nov 04 00:21:47.898279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.898187    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273"} err="failed to get container status \"4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273\": rpc error: code = NotFound desc = could not find container \"4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273\": container with ID starting with 4c0db4029a8dc7591df02ed24f03299174822917e1dec4cfb4d539898c501273 not found: ID does not exist"
Nov 04 00:21:47.898279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.898205    5074 scope.go:122] "RemoveContainer" containerID="5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d"
Nov 04 00:21:47.911706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.911683    5074 scope.go:122] "RemoveContainer" containerID="5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d"
Nov 04 00:21:47.912028 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:21:47.912004    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d\": container with ID starting with 5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d not found: ID does not exist" containerID="5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d"
Nov 04 00:21:47.912098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.912039    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d"} err="failed to get container status \"5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d\": rpc error: code = NotFound desc = could not find container \"5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d\": container with ID starting with 5be14167f986723782eda36e353abbe0f101989fa012500098b0fba364322a2d not found: ID does not exist"
Nov 04 00:21:47.912098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.912063    5074 scope.go:122] "RemoveContainer" containerID="6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57"
Nov 04 00:21:47.921068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.921019    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"]
Nov 04 00:21:47.928507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.928484    5074 scope.go:122] "RemoveContainer" containerID="6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57"
Nov 04 00:21:47.928864 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:21:47.928841    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57\": container with ID starting with 6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57 not found: ID does not exist" containerID="6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57"
Nov 04 00:21:47.928953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.928875    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57"} err="failed to get container status \"6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57\": rpc error: code = NotFound desc = could not find container \"6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57\": container with ID starting with 6655a4ac3015e9a78afe1d8d591273bac4a33073fe5c3e0c8598e6cc2cadbd57 not found: ID does not exist"
Nov 04 00:21:47.929090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.929067    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8s5gs"]
Nov 04 00:21:47.955467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955421    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.955467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955467    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955481    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955496    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955509    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955523    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88bqs\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-kube-api-access-88bqs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955539    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955606    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955657    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955670    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955684    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955699    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955714    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxz6n\" (UniqueName: \"kubernetes.io/projected/d16665b8-d193-41b7-a0b4-32108cbf6a6f-kube-api-access-sxz6n\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955728    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955756    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955776    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955790    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6lt\" (UniqueName: \"kubernetes.io/projected/324fe1c5-df49-4e36-92d0-f4dccead361a-kube-api-access-7h6lt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955805    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955819    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324fe1c5-df49-4e36-92d0-f4dccead361a-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955838    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324fe1c5-df49-4e36-92d0-f4dccead361a-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955853    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16665b8-d193-41b7-a0b4-32108cbf6a6f-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955867    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955881    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955894    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.956071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:47.955908    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16665b8-d193-41b7-a0b4-32108cbf6a6f-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:48.403003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:48.402601    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"]
Nov 04 00:21:48.430818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:48.430785    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-5tbzq"]
Nov 04 00:21:48.435969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:48.435939    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"]
Nov 04 00:21:48.441149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:48.441125    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ln7lf"]
Nov 04 00:21:49.020694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:49.020662    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" path="/var/lib/kubelet/pods/324fe1c5-df49-4e36-92d0-f4dccead361a/volumes"
Nov 04 00:21:49.021303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:49.021285    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" path="/var/lib/kubelet/pods/4f32ec71-4f6a-49c4-9ba5-f6c931e159d7/volumes"
Nov 04 00:21:49.021782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:21:49.021765    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" path="/var/lib/kubelet/pods/d16665b8-d193-41b7-a0b4-32108cbf6a6f/volumes"
Nov 04 00:21:58.026640 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:21:58.026594    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:22:00.333978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:00.333941    5074 scope.go:122] "RemoveContainer" containerID="8d2d212610a2e06e834878103dd3a27ddba2cbfa175947bd8417b746c105b71e"
Nov 04 00:22:00.347252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:00.347222    5074 scope.go:122] "RemoveContainer" containerID="a11dbdf76babf89ad19fa77d895be8003ea534fa643908c458a06c2f389450fa"
Nov 04 00:22:00.358410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:00.358385    5074 scope.go:122] "RemoveContainer" containerID="b167296645c11a6e8d71618f31c9bec905dbe28dd5923982d4e76da97bae85d9"
Nov 04 00:22:00.369339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:00.369318    5074 scope.go:122] "RemoveContainer" containerID="75456528775b57caed04c6d7a54456af4fb50d2dab2459bbe29a22f6513e6ee2"
Nov 04 00:22:03.592069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.592036    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"]
Nov 04 00:22:03.593077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593058    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy"
Nov 04 00:22:03.593077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593078    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy"
Nov 04 00:22:03.593165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593113    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy"
Nov 04 00:22:03.593165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593118    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy"
Nov 04 00:22:03.593165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593135    5074 cpu_manager.go:414] "containerMap: removing container" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy"
Nov 04 00:22:03.593165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593140    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy"
Nov 04 00:22:03.593294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593283    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16665b8-d193-41b7-a0b4-32108cbf6a6f" containerName="istio-proxy"
Nov 04 00:22:03.593334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593296    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="324fe1c5-df49-4e36-92d0-f4dccead361a" containerName="istio-proxy"
Nov 04 00:22:03.593334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.593311    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f32ec71-4f6a-49c4-9ba5-f6c931e159d7" containerName="istio-proxy"
Nov 04 00:22:03.599211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.599186    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.606732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.606705    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"]
Nov 04 00:22:03.608255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.608233    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:22:03.608515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.608475    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:22:03.648404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.648368    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.648404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.648405    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.648627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.648497    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqtw\" (UniqueName: \"kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.750342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.750280    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.750540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.750351    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.750540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.750413    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqtw\" (UniqueName: \"kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.751699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.751659    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.753717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.753693    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.763386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.763357    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqtw\" (UniqueName: \"kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw\") pod \"maas-api-revoke-sub-6ca0a119-tfnc9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:03.930291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:03.930210    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:04.224999 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:22:04.224958    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe991d81_1fd3_4c71_8895_ec5c34d163d9.slice/crio-62364a9a3ade86e6e1689d5e2432234b516bbbcf500f228609db0fbec177b608 WatchSource:0}: Error finding container 62364a9a3ade86e6e1689d5e2432234b516bbbcf500f228609db0fbec177b608: Status 404 returned error can't find the container with id 62364a9a3ade86e6e1689d5e2432234b516bbbcf500f228609db0fbec177b608
Nov 04 00:22:04.225507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:04.225471    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"]
Nov 04 00:22:05.071184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:05.071142    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9" event={"ID":"be991d81-1fd3-4c71-8895-ec5c34d163d9","Type":"ContainerStarted","Data":"595065dbdf146fef549c9299edd51a4df9f5f7dc4ca6524496ce24a5e2945301"}
Nov 04 00:22:05.071596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:05.071202    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9" event={"ID":"be991d81-1fd3-4c71-8895-ec5c34d163d9","Type":"ContainerStarted","Data":"62364a9a3ade86e6e1689d5e2432234b516bbbcf500f228609db0fbec177b608"}
Nov 04 00:22:05.092585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:05.092530    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9" podStartSLOduration=2.09251682 podStartE2EDuration="2.09251682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:22:05.089764875 +0000 UTC m=+3190.189365009" watchObservedRunningTime="2026-11-04 00:22:05.09251682 +0000 UTC m=+3190.192116964"
Nov 04 00:22:06.084443 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:06.084408    5074 generic.go:356] "Generic (PLEG): container finished" podID="be991d81-1fd3-4c71-8895-ec5c34d163d9" containerID="595065dbdf146fef549c9299edd51a4df9f5f7dc4ca6524496ce24a5e2945301" exitCode=0
Nov 04 00:22:06.084859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:06.084466    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9" event={"ID":"be991d81-1fd3-4c71-8895-ec5c34d163d9","Type":"ContainerDied","Data":"595065dbdf146fef549c9299edd51a4df9f5f7dc4ca6524496ce24a5e2945301"}
Nov 04 00:22:07.374174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.374110    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:07.428063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.428027    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw\" (UniqueName: \"kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw\") pod \"be991d81-1fd3-4c71-8895-ec5c34d163d9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") "
Nov 04 00:22:07.428380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.428358    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca\") pod \"be991d81-1fd3-4c71-8895-ec5c34d163d9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") "
Nov 04 00:22:07.428484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.428410    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca\") pod \"be991d81-1fd3-4c71-8895-ec5c34d163d9\" (UID: \"be991d81-1fd3-4c71-8895-ec5c34d163d9\") "
Nov 04 00:22:07.429841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.429810    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca" pod "be991d81-1fd3-4c71-8895-ec5c34d163d9" (UID: "be991d81-1fd3-4c71-8895-ec5c34d163d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:22:07.432302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.432273    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw" pod "be991d81-1fd3-4c71-8895-ec5c34d163d9" (UID: "be991d81-1fd3-4c71-8895-ec5c34d163d9"). InnerVolumeSpecName "kube-api-access-2nqtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:07.432429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.432319    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca" pod "be991d81-1fd3-4c71-8895-ec5c34d163d9" (UID: "be991d81-1fd3-4c71-8895-ec5c34d163d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:07.530645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.530609    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:07.530645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.530642    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be991d81-1fd3-4c71-8895-ec5c34d163d9-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:07.530645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:07.530654    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqtw\" (UniqueName: \"kubernetes.io/projected/be991d81-1fd3-4c71-8895-ec5c34d163d9-kube-api-access-2nqtw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:08.115417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:08.115382    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"
Nov 04 00:22:08.115590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:08.115394    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9" event={"ID":"be991d81-1fd3-4c71-8895-ec5c34d163d9","Type":"ContainerDied","Data":"62364a9a3ade86e6e1689d5e2432234b516bbbcf500f228609db0fbec177b608"}
Nov 04 00:22:08.115590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:08.115514    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62364a9a3ade86e6e1689d5e2432234b516bbbcf500f228609db0fbec177b608"
Nov 04 00:22:14.295435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.295399    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"]
Nov 04 00:22:14.811687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.811640    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"]
Nov 04 00:22:14.813319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.813291    5074 cpu_manager.go:414] "containerMap: removing container" podUID="be991d81-1fd3-4c71-8895-ec5c34d163d9" containerName="revoke-keys"
Nov 04 00:22:14.813319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.813318    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be991d81-1fd3-4c71-8895-ec5c34d163d9" containerName="revoke-keys"
Nov 04 00:22:14.813622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.813606    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="be991d81-1fd3-4c71-8895-ec5c34d163d9" containerName="revoke-keys"
Nov 04 00:22:14.833102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.833070    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"]
Nov 04 00:22:14.833405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.833386    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:14.844311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.844288    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:22:14.844595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.844578    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:22:14.955009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.954973    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6st6\" (UniqueName: \"kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:14.955184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.955025    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:14.955184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:14.955107    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.056866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.056830    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6st6\" (UniqueName: \"kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.057063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.056880    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.057063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.056911    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.057610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.057581    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.059538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.059511    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.089854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.089574    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6st6\" (UniqueName: \"kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6\") pod \"maas-api-revoke-sub-85b4fc32-7jzxx\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.157842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.157800    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:15.455267 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:22:15.455232    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7e4d27_0ada_44d6_989a_098a33b5a4f4.slice/crio-6fc1927998d7113ab26a2ce1341a9321f1ae3fea8a4a3ac876e6929a279e70f8 WatchSource:0}: Error finding container 6fc1927998d7113ab26a2ce1341a9321f1ae3fea8a4a3ac876e6929a279e70f8: Status 404 returned error can't find the container with id 6fc1927998d7113ab26a2ce1341a9321f1ae3fea8a4a3ac876e6929a279e70f8
Nov 04 00:22:15.458651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:15.458521    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"]
Nov 04 00:22:16.204225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:16.204179    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx" event={"ID":"ba7e4d27-0ada-44d6-989a-098a33b5a4f4","Type":"ContainerStarted","Data":"b35da35267f699b70f845681fa173040a975ffa540336869636e3d9d826d220f"}
Nov 04 00:22:16.204430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:16.204288    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx" event={"ID":"ba7e4d27-0ada-44d6-989a-098a33b5a4f4","Type":"ContainerStarted","Data":"6fc1927998d7113ab26a2ce1341a9321f1ae3fea8a4a3ac876e6929a279e70f8"}
Nov 04 00:22:16.227876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:16.227801    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx" podStartSLOduration=2.227781823 podStartE2EDuration="2.227781823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:22:16.224957011 +0000 UTC m=+3201.324557156" watchObservedRunningTime="2026-11-04 00:22:16.227781823 +0000 UTC m=+3201.327381970"
Nov 04 00:22:17.218962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:17.218920    5074 generic.go:356] "Generic (PLEG): container finished" podID="ba7e4d27-0ada-44d6-989a-098a33b5a4f4" containerID="b35da35267f699b70f845681fa173040a975ffa540336869636e3d9d826d220f" exitCode=0
Nov 04 00:22:17.219591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:17.218964    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx" event={"ID":"ba7e4d27-0ada-44d6-989a-098a33b5a4f4","Type":"ContainerDied","Data":"b35da35267f699b70f845681fa173040a975ffa540336869636e3d9d826d220f"}
Nov 04 00:22:18.798140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.798111    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:18.909428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.909392    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6\" (UniqueName: \"kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6\") pod \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") "
Nov 04 00:22:18.909640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.909562    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca\") pod \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") "
Nov 04 00:22:18.909704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.909686    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca\") pod \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\" (UID: \"ba7e4d27-0ada-44d6-989a-098a33b5a4f4\") "
Nov 04 00:22:18.910244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.910210    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca" pod "ba7e4d27-0ada-44d6-989a-098a33b5a4f4" (UID: "ba7e4d27-0ada-44d6-989a-098a33b5a4f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:22:18.910537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.910513    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:18.912324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.912300    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6" pod "ba7e4d27-0ada-44d6-989a-098a33b5a4f4" (UID: "ba7e4d27-0ada-44d6-989a-098a33b5a4f4"). InnerVolumeSpecName "kube-api-access-k6st6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:18.912408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:18.912313    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca" pod "ba7e4d27-0ada-44d6-989a-098a33b5a4f4" (UID: "ba7e4d27-0ada-44d6-989a-098a33b5a4f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:19.039817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:19.037225    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6st6\" (UniqueName: \"kubernetes.io/projected/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-kube-api-access-k6st6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:19.039817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:19.037265    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7e4d27-0ada-44d6-989a-098a33b5a4f4-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:19.277370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:19.277336    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"
Nov 04 00:22:19.277560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:19.277375    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx" event={"ID":"ba7e4d27-0ada-44d6-989a-098a33b5a4f4","Type":"ContainerDied","Data":"6fc1927998d7113ab26a2ce1341a9321f1ae3fea8a4a3ac876e6929a279e70f8"}
Nov 04 00:22:19.277560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:19.277427    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fc1927998d7113ab26a2ce1341a9321f1ae3fea8a4a3ac876e6929a279e70f8"
Nov 04 00:22:28.038416 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:22:28.038374    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:22:29.328402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.328308    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" podUID="0f98430b-3385-4f3a-8d70-807e6b5b0825" containerName="main" containerID="cri-o://7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f" gracePeriod=45
Nov 04 00:22:29.437566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.437526    5074 generic.go:356] "Generic (PLEG): container finished" podID="113595e4-c554-4735-8d85-6e719e9b8c24" containerID="b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b" exitCode=0
Nov 04 00:22:29.437814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.437582    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" event={"ID":"113595e4-c554-4735-8d85-6e719e9b8c24","Type":"ContainerDied","Data":"b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"}
Nov 04 00:22:29.438188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.438144    5074 scope.go:122] "RemoveContainer" containerID="b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"
Nov 04 00:22:29.440929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.440788    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:22:29.440929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.440862    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:22:29.440929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.440896    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:22:29.775644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.775613    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:22:29.868539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.868440    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm\") pod \"0f98430b-3385-4f3a-8d70-807e6b5b0825\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") "
Nov 04 00:22:29.868756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.868583    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs\") pod \"0f98430b-3385-4f3a-8d70-807e6b5b0825\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") "
Nov 04 00:22:29.868756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.868712    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir\") pod \"0f98430b-3385-4f3a-8d70-807e6b5b0825\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") "
Nov 04 00:22:29.868900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.868764    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b\" (UniqueName: \"kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b\") pod \"0f98430b-3385-4f3a-8d70-807e6b5b0825\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") "
Nov 04 00:22:29.868900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.868874    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home\") pod \"0f98430b-3385-4f3a-8d70-807e6b5b0825\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") "
Nov 04 00:22:29.869005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.868930    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache\") pod \"0f98430b-3385-4f3a-8d70-807e6b5b0825\" (UID: \"0f98430b-3385-4f3a-8d70-807e6b5b0825\") "
Nov 04 00:22:29.869621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.869586    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir" pod "0f98430b-3385-4f3a-8d70-807e6b5b0825" (UID: "0f98430b-3385-4f3a-8d70-807e6b5b0825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:29.870389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.870359    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home" pod "0f98430b-3385-4f3a-8d70-807e6b5b0825" (UID: "0f98430b-3385-4f3a-8d70-807e6b5b0825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:29.870577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.870556    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache" pod "0f98430b-3385-4f3a-8d70-807e6b5b0825" (UID: "0f98430b-3385-4f3a-8d70-807e6b5b0825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:29.878133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.877251    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm" pod "0f98430b-3385-4f3a-8d70-807e6b5b0825" (UID: "0f98430b-3385-4f3a-8d70-807e6b5b0825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:29.878133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.877931    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b" pod "0f98430b-3385-4f3a-8d70-807e6b5b0825" (UID: "0f98430b-3385-4f3a-8d70-807e6b5b0825"). InnerVolumeSpecName "kube-api-access-m8p5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:29.878875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.878850    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs" pod "0f98430b-3385-4f3a-8d70-807e6b5b0825" (UID: "0f98430b-3385-4f3a-8d70-807e6b5b0825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:29.973658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.972501    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:29.973658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.972573    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f98430b-3385-4f3a-8d70-807e6b5b0825-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:29.973658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.972593    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:29.973658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.972610    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8p5b\" (UniqueName: \"kubernetes.io/projected/0f98430b-3385-4f3a-8d70-807e6b5b0825-kube-api-access-m8p5b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:29.973658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.972647    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:29.973658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:29.972663    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f98430b-3385-4f3a-8d70-807e6b5b0825-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:30.452091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.452046    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" event={"ID":"113595e4-c554-4735-8d85-6e719e9b8c24","Type":"ContainerStarted","Data":"8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de"}
Nov 04 00:22:30.453795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.453724    5074 generic.go:356] "Generic (PLEG): container finished" podID="0f98430b-3385-4f3a-8d70-807e6b5b0825" containerID="7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f" exitCode=0
Nov 04 00:22:30.453940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.453807    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"
Nov 04 00:22:30.453940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.453812    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" event={"ID":"0f98430b-3385-4f3a-8d70-807e6b5b0825","Type":"ContainerDied","Data":"7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f"}
Nov 04 00:22:30.453940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.453929    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" event={"ID":"0f98430b-3385-4f3a-8d70-807e6b5b0825","Type":"ContainerDied","Data":"43ffff5746c856f23ce087acb833480ffc5f0614d01c5c27dc169bcefff2ab18"}
Nov 04 00:22:30.454098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.453949    5074 scope.go:122] "RemoveContainer" containerID="7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f"
Nov 04 00:22:30.466955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.466933    5074 scope.go:122] "RemoveContainer" containerID="7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f"
Nov 04 00:22:30.467263 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:22:30.467239    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f\": container with ID starting with 7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f not found: ID does not exist" containerID="7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f"
Nov 04 00:22:30.467330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.467270    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f"} err="failed to get container status \"7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f\": rpc error: code = NotFound desc = could not find container \"7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f\": container with ID starting with 7a02f9fa802901558554d2aaeea637a66f16d73a8c4dc44b898fc07344e4946f not found: ID does not exist"
Nov 04 00:22:30.503658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.503627    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"]
Nov 04 00:22:30.511672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.511641    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb"]
Nov 04 00:22:30.620763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:30.617120    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-11301e/test-model-11301e-kserve-6676d87b69-htplb" podUID="0f98430b-3385-4f3a-8d70-807e6b5b0825" containerName="main" probeResult="failure" output="Get \"https://10.132.1.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:22:31.069311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:31.069268    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f98430b-3385-4f3a-8d70-807e6b5b0825" path="/var/lib/kubelet/pods/0f98430b-3385-4f3a-8d70-807e6b5b0825/volumes"
Nov 04 00:22:31.296235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:31.296189    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:22:31.297411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:31.297384    5074 patch_prober.go:29] interesting pod/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body=
Nov 04 00:22:31.297573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:31.297437    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused"
Nov 04 00:22:32.048818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:32.042714    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"]
Nov 04 00:22:32.053619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:32.053586    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6423ae74-tq67v"]
Nov 04 00:22:32.295487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:32.295445    5074 patch_prober.go:29] interesting pod/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body=
Nov 04 00:22:32.295673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:32.295505    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused"
Nov 04 00:22:33.039249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:33.039209    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74882772-cb33-45ea-96f8-018b402b034b" path="/var/lib/kubelet/pods/74882772-cb33-45ea-96f8-018b402b034b/volumes"
Nov 04 00:22:33.068793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:33.068762    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:22:33.296175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:33.296092    5074 patch_prober.go:29] interesting pod/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body=
Nov 04 00:22:33.296175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:33.296151    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused"
Nov 04 00:22:34.304773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:34.304088    5074 patch_prober.go:29] interesting pod/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body=
Nov 04 00:22:34.304773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:34.304166    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused"
Nov 04 00:22:35.095909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:35.095872    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"]
Nov 04 00:22:35.147815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:35.147174    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"]
Nov 04 00:22:35.297222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:35.296562    5074 patch_prober.go:29] interesting pod/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body=
Nov 04 00:22:35.297222 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:35.296635    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused"
Nov 04 00:22:36.295558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:36.295509    5074 patch_prober.go:29] interesting pod/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body=
Nov 04 00:22:36.296176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:36.295572    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused"
Nov 04 00:22:37.346174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:37.344300    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:22:37.347578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:37.347554    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:22:40.099843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:40.097036    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" containerName="payload-processing" containerID="cri-o://d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879" gracePeriod=25
Nov 04 00:22:40.148115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:40.148046    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerName="payload-pre-processing" containerID="cri-o://c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c" gracePeriod=25
Nov 04 00:22:42.072874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:42.072822    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"]
Nov 04 00:22:42.075529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:42.075488    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e44247-nhrdn"]
Nov 04 00:22:43.024866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:43.024827    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c372e3a3-2e12-49b2-bbba-c8d15319ce32" path="/var/lib/kubelet/pods/c372e3a3-2e12-49b2-bbba-c8d15319ce32/volumes"
Nov 04 00:22:45.478900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:45.478857    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:22:45.479359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:45.479156    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:22:52.063708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:52.063666    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"]
Nov 04 00:22:52.070385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:52.070349    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"]
Nov 04 00:22:55.479213 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:55.479176    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:22:55.479616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:55.479264    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:22:57.064937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:57.064895    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" containerName="payload-processing" containerID="cri-o://b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e" gracePeriod=25
Nov 04 00:22:57.071193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:57.071163    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" containerName="payload-pre-processing" containerID="cri-o://46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce" gracePeriod=25
Nov 04 00:22:58.023847 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:22:58.023801    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:22:58.857217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:58.857182    5074 generic.go:356] "Generic (PLEG): container finished" podID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerID="27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47" exitCode=1
Nov 04 00:22:58.857632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:58.857256    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" event={"ID":"07f9e9ff-13f0-4d3c-ba08-a308ce67d227","Type":"ContainerDied","Data":"27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"}
Nov 04 00:22:58.857706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:58.857661    5074 scope.go:122] "RemoveContainer" containerID="27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"
Nov 04 00:22:59.365991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:59.365968    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 04 00:22:59.870086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:59.870043    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" event={"ID":"07f9e9ff-13f0-4d3c-ba08-a308ce67d227","Type":"ContainerStarted","Data":"4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc"}
Nov 04 00:22:59.870579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:22:59.870293    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 04 00:23:00.484037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.484008    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:23:00.599771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.598963    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k\" (UniqueName: \"kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k\") pod \"0a897f01-7e97-40d6-9e16-70deeede895f\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") "
Nov 04 00:23:00.599771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.599262    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume\") pod \"0a897f01-7e97-40d6-9e16-70deeede895f\" (UID: \"0a897f01-7e97-40d6-9e16-70deeede895f\") "
Nov 04 00:23:00.618762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.605882    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume" pod "0a897f01-7e97-40d6-9e16-70deeede895f" (UID: "0a897f01-7e97-40d6-9e16-70deeede895f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:00.653048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.647200    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a897f01-7e97-40d6-9e16-70deeede895f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:00.653048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.647803    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k" pod "0a897f01-7e97-40d6-9e16-70deeede895f" (UID: "0a897f01-7e97-40d6-9e16-70deeede895f"). InnerVolumeSpecName "kube-api-access-s652k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:00.677764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.666603    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:23:00.769789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.760926    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl\" (UniqueName: \"kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl\") pod \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") "
Nov 04 00:23:00.769789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.761117    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume\") pod \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\" (UID: \"7f0c9be1-08ce-4e46-bc72-1605dce70b7d\") "
Nov 04 00:23:00.769789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.762017    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s652k\" (UniqueName: \"kubernetes.io/projected/0a897f01-7e97-40d6-9e16-70deeede895f-kube-api-access-s652k\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:00.786198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.780309    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume" pod "7f0c9be1-08ce-4e46-bc72-1605dce70b7d" (UID: "7f0c9be1-08ce-4e46-bc72-1605dce70b7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:00.786198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.786076    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl" pod "7f0c9be1-08ce-4e46-bc72-1605dce70b7d" (UID: "7f0c9be1-08ce-4e46-bc72-1605dce70b7d"). InnerVolumeSpecName "kube-api-access-m7tpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:00.863930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.863894    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tpl\" (UniqueName: \"kubernetes.io/projected/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-kube-api-access-m7tpl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:00.863930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.863924    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0c9be1-08ce-4e46-bc72-1605dce70b7d-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:00.880883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.880847    5074 generic.go:356] "Generic (PLEG): container finished" podID="0a897f01-7e97-40d6-9e16-70deeede895f" containerID="d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879" exitCode=0
Nov 04 00:23:00.881345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.880894    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"
Nov 04 00:23:00.881345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.880927    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" event={"ID":"0a897f01-7e97-40d6-9e16-70deeede895f","Type":"ContainerDied","Data":"d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879"}
Nov 04 00:23:00.881345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.880994    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c" event={"ID":"0a897f01-7e97-40d6-9e16-70deeede895f","Type":"ContainerDied","Data":"bec270c16f9a0cb38809f025147636f0b49e5c0e7ee5bb5102f82818451fef1d"}
Nov 04 00:23:00.881345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.881018    5074 scope.go:122] "RemoveContainer" containerID="d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879"
Nov 04 00:23:00.883170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.883133    5074 generic.go:356] "Generic (PLEG): container finished" podID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerID="c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c" exitCode=0
Nov 04 00:23:00.883350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.883333    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"
Nov 04 00:23:00.883505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.883373    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" event={"ID":"7f0c9be1-08ce-4e46-bc72-1605dce70b7d","Type":"ContainerDied","Data":"c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c"}
Nov 04 00:23:00.883505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.883426    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb" event={"ID":"7f0c9be1-08ce-4e46-bc72-1605dce70b7d","Type":"ContainerDied","Data":"855acbcdbc4c5f98d40a7514e7f0b89b9f6b4d0c9c70eff27bd451d0b3f08fe3"}
Nov 04 00:23:00.897716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.897696    5074 scope.go:122] "RemoveContainer" containerID="d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879"
Nov 04 00:23:00.899024 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:00.898875    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879\": container with ID starting with d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879 not found: ID does not exist" containerID="d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879"
Nov 04 00:23:00.899024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.898913    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879"} err="failed to get container status \"d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879\": rpc error: code = NotFound desc = could not find container \"d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879\": container with ID starting with d5477f1d1b94d71bae5ffe870d9652095dd5d790fa5da496df27ac9f2356d879 not found: ID does not exist"
Nov 04 00:23:00.899024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.898937    5074 scope.go:122] "RemoveContainer" containerID="c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c"
Nov 04 00:23:00.914248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.914071    5074 scope.go:122] "RemoveContainer" containerID="c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c"
Nov 04 00:23:00.914568 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:00.914540    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c\": container with ID starting with c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c not found: ID does not exist" containerID="c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c"
Nov 04 00:23:00.914675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.914590    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c"} err="failed to get container status \"c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c\": rpc error: code = NotFound desc = could not find container \"c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c\": container with ID starting with c6302add4aa1528f290cc515bdda5f27cfb1a80002c5e254648d44b5419da85c not found: ID does not exist"
Nov 04 00:23:00.921884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.921827    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"]
Nov 04 00:23:00.925907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.925877    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11301e-7db6f89454-4sk5c"]
Nov 04 00:23:00.940901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.940870    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"]
Nov 04 00:23:00.949530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:00.949498    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11301e-65586d55cf-68npb"]
Nov 04 00:23:01.024996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:01.024961    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" path="/var/lib/kubelet/pods/0a897f01-7e97-40d6-9e16-70deeede895f/volumes"
Nov 04 00:23:01.025476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:01.025460    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" path="/var/lib/kubelet/pods/7f0c9be1-08ce-4e46-bc72-1605dce70b7d/volumes"
Nov 04 00:23:01.676606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:01.676566    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:23:02.696971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:02.696931    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:23:10.885977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:10.885937    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 04 00:23:11.677168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:11.677130    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:23:12.697000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:12.696964    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:23:17.423560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.423160    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:23:17.457112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.457085    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:23:17.458232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.458206    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume\") pod \"1be8c401-cee0-4195-b71e-c76b29f03954\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") "
Nov 04 00:23:17.461430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.461402    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m\" (UniqueName: \"kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m\") pod \"1be8c401-cee0-4195-b71e-c76b29f03954\" (UID: \"1be8c401-cee0-4195-b71e-c76b29f03954\") "
Nov 04 00:23:17.462540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.460967    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume" pod "1be8c401-cee0-4195-b71e-c76b29f03954" (UID: "1be8c401-cee0-4195-b71e-c76b29f03954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:17.465309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.465278    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m" pod "1be8c401-cee0-4195-b71e-c76b29f03954" (UID: "1be8c401-cee0-4195-b71e-c76b29f03954"). InnerVolumeSpecName "kube-api-access-qjx6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:17.563061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.563028    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br\" (UniqueName: \"kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br\") pod \"1138f268-e37d-40a9-8214-9b7af5714a37\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") "
Nov 04 00:23:17.563220 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.563077    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume\") pod \"1138f268-e37d-40a9-8214-9b7af5714a37\" (UID: \"1138f268-e37d-40a9-8214-9b7af5714a37\") "
Nov 04 00:23:17.563558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.563522    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume" pod "1138f268-e37d-40a9-8214-9b7af5714a37" (UID: "1138f268-e37d-40a9-8214-9b7af5714a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:17.563700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.563686    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1138f268-e37d-40a9-8214-9b7af5714a37-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:17.563765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.563703    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be8c401-cee0-4195-b71e-c76b29f03954-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:17.563765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.563715    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjx6m\" (UniqueName: \"kubernetes.io/projected/1be8c401-cee0-4195-b71e-c76b29f03954-kube-api-access-qjx6m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:17.565157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.565135    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br" pod "1138f268-e37d-40a9-8214-9b7af5714a37" (UID: "1138f268-e37d-40a9-8214-9b7af5714a37"). InnerVolumeSpecName "kube-api-access-d94br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:17.665452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:17.665368    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94br\" (UniqueName: \"kubernetes.io/projected/1138f268-e37d-40a9-8214-9b7af5714a37-kube-api-access-d94br\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:18.143581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.143531    5074 generic.go:356] "Generic (PLEG): container finished" podID="1138f268-e37d-40a9-8214-9b7af5714a37" containerID="b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e" exitCode=0
Nov 04 00:23:18.143581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.143582    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"
Nov 04 00:23:18.143976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.143634    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" event={"ID":"1138f268-e37d-40a9-8214-9b7af5714a37","Type":"ContainerDied","Data":"b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e"}
Nov 04 00:23:18.143976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.143718    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c" event={"ID":"1138f268-e37d-40a9-8214-9b7af5714a37","Type":"ContainerDied","Data":"04d1a910ddcdaf7748922e1a6b4d88f6e1874938a24c7c75856fc544f33420d2"}
Nov 04 00:23:18.143976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.143770    5074 scope.go:122] "RemoveContainer" containerID="b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e"
Nov 04 00:23:18.145593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.145565    5074 generic.go:356] "Generic (PLEG): container finished" podID="1be8c401-cee0-4195-b71e-c76b29f03954" containerID="46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce" exitCode=0
Nov 04 00:23:18.145707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.145607    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" event={"ID":"1be8c401-cee0-4195-b71e-c76b29f03954","Type":"ContainerDied","Data":"46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce"}
Nov 04 00:23:18.145707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.145638    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8" event={"ID":"1be8c401-cee0-4195-b71e-c76b29f03954","Type":"ContainerDied","Data":"82e218e958027710b52598b6d833ee2afa50197f0b843d693d4c5ecccb60b007"}
Nov 04 00:23:18.145707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.145692    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"
Nov 04 00:23:18.158923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.158901    5074 scope.go:122] "RemoveContainer" containerID="b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e"
Nov 04 00:23:18.159268 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:18.159243    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e\": container with ID starting with b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e not found: ID does not exist" containerID="b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e"
Nov 04 00:23:18.159316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.159276    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e"} err="failed to get container status \"b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e\": rpc error: code = NotFound desc = could not find container \"b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e\": container with ID starting with b6bf2197083aac49ef78631cffb605e32c674199f70d5b8c763e990105b71e7e not found: ID does not exist"
Nov 04 00:23:18.159316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.159292    5074 scope.go:122] "RemoveContainer" containerID="46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce"
Nov 04 00:23:18.172387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.172364    5074 scope.go:122] "RemoveContainer" containerID="46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce"
Nov 04 00:23:18.172670 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:18.172647    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce\": container with ID starting with 46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce not found: ID does not exist" containerID="46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce"
Nov 04 00:23:18.172764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.172677    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce"} err="failed to get container status \"46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce\": rpc error: code = NotFound desc = could not find container \"46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce\": container with ID starting with 46f8dddb1d88a2c09d8189cef1f077a2d53f443894ec7d218afeb25f2a46d8ce not found: ID does not exist"
Nov 04 00:23:18.174197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.174173    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"]
Nov 04 00:23:18.182244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.182214    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cc1c0-77fb49d9c8-4gf2c"]
Nov 04 00:23:18.198693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.198624    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"]
Nov 04 00:23:18.203514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:18.203486    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cc1c0-5d487c7d99-bpvk8"]
Nov 04 00:23:19.022785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:19.022734    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" path="/var/lib/kubelet/pods/1138f268-e37d-40a9-8214-9b7af5714a37/volumes"
Nov 04 00:23:19.023458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:19.023438    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" path="/var/lib/kubelet/pods/1be8c401-cee0-4195-b71e-c76b29f03954/volumes"
Nov 04 00:23:28.025986 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:28.025942    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:23:35.029654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:35.029618    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"]
Nov 04 00:23:35.034300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:35.034271    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7956f4e8-wbvjg"]
Nov 04 00:23:35.680763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:35.675986    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"]
Nov 04 00:23:35.680763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:35.676302    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy" containerID="cri-o://25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508" gracePeriod=30
Nov 04 00:23:35.696732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:35.696663    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"]
Nov 04 00:23:35.697207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:35.697174    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy" containerID="cri-o://2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572" gracePeriod=30
Nov 04 00:23:37.022059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:37.022022    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e17594-e4b1-4c74-9638-7dd2509a0e64" path="/var/lib/kubelet/pods/55e17594-e4b1-4c74-9638-7dd2509a0e64/volumes"
Nov 04 00:23:41.059188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.059119    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:23:41.094838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.094810    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:23:41.187437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187392    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.187639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187460    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.187639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187545    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.187639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187605    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.187867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187653    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.187867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187699    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.187867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187811    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.187867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187839    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.187867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187850    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.188153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187947    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.188153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.187984    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.188153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188048    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.188153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188141    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.188383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188164    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.188383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188188    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.188383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188226    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.188383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188255    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.188614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188397    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.188614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188426    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket\") pod \"9f45a4f9-10f2-43f3-bda6-492500f39719\" (UID: \"9f45a4f9-10f2-43f3-bda6-492500f39719\") "
Nov 04 00:23:41.188614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.188482    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data\") pod \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\" (UID: \"9f62472d-3985-44e3-ade7-a2a87d28ffcb\") "
Nov 04 00:23:41.189124 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.189092    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.189351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.189329    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.190152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.189855    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:41.191616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.191591    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.192040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.192007    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.193315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.192298    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:41.193315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.192553    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.193315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.192733    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193331    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193353    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193369    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193385    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193401    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193415    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193430    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193444    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193459    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45a4f9-10f2-43f3-bda6-492500f39719-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.193767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.193472    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.195669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.195645    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:23:41.196102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.196079    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.196179 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.196097    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:23:41.196296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.196273    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.196332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.196308    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7" pod "9f45a4f9-10f2-43f3-bda6-492500f39719" (UID: "9f45a4f9-10f2-43f3-bda6-492500f39719"). InnerVolumeSpecName "kube-api-access-dllq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.198125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.197998    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.198272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.198247    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.198592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.198558    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h" pod "9f62472d-3985-44e3-ade7-a2a87d28ffcb" (UID: "9f62472d-3985-44e3-ade7-a2a87d28ffcb"). InnerVolumeSpecName "kube-api-access-rbr9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295583    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllq7\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-kube-api-access-dllq7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295617    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbr9h\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-kube-api-access-rbr9h\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295628    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295637    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295646    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295654    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45a4f9-10f2-43f3-bda6-492500f39719-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295663    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.295670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.295672    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f62472d-3985-44e3-ade7-a2a87d28ffcb-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.474555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.474519    5074 generic.go:356] "Generic (PLEG): container finished" podID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerID="25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508" exitCode=0
Nov 04 00:23:41.474794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.474570    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"
Nov 04 00:23:41.474794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.474599    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" event={"ID":"9f62472d-3985-44e3-ade7-a2a87d28ffcb","Type":"ContainerDied","Data":"25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508"}
Nov 04 00:23:41.474794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.474656    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b" event={"ID":"9f62472d-3985-44e3-ade7-a2a87d28ffcb","Type":"ContainerDied","Data":"5fcf091c994136bd0c2be3711473a2a9d1d106b925de3b1983ebd3f06601641b"}
Nov 04 00:23:41.474794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.474685    5074 scope.go:122] "RemoveContainer" containerID="25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508"
Nov 04 00:23:41.476578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.476555    5074 generic.go:356] "Generic (PLEG): container finished" podID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerID="2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572" exitCode=0
Nov 04 00:23:41.476694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.476612    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"
Nov 04 00:23:41.476694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.476650    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" event={"ID":"9f45a4f9-10f2-43f3-bda6-492500f39719","Type":"ContainerDied","Data":"2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572"}
Nov 04 00:23:41.476845 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.476723    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh" event={"ID":"9f45a4f9-10f2-43f3-bda6-492500f39719","Type":"ContainerDied","Data":"c03e6be9a3c7b5cb849d2897f6c61ed2113f8622a495a9c2a6bbfc1348ef5751"}
Nov 04 00:23:41.488142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.488126    5074 scope.go:122] "RemoveContainer" containerID="25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508"
Nov 04 00:23:41.488405 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:41.488386    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508\": container with ID starting with 25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508 not found: ID does not exist" containerID="25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508"
Nov 04 00:23:41.488493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.488416    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508"} err="failed to get container status \"25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508\": rpc error: code = NotFound desc = could not find container \"25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508\": container with ID starting with 25fbf450147e4478c78b96f4029b1c6c9a94724606e439ff18a5be54b43e0508 not found: ID does not exist"
Nov 04 00:23:41.488493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.488440    5074 scope.go:122] "RemoveContainer" containerID="2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572"
Nov 04 00:23:41.500242 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.500215    5074 scope.go:122] "RemoveContainer" containerID="2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572"
Nov 04 00:23:41.500577 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:41.500555    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572\": container with ID starting with 2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572 not found: ID does not exist" containerID="2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572"
Nov 04 00:23:41.500660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.500587    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572"} err="failed to get container status \"2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572\": rpc error: code = NotFound desc = could not find container \"2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572\": container with ID starting with 2832456b402a63eca5d0c158401bda3d1958a7fa2929a54d4705d9f3e29cd572 not found: ID does not exist"
Nov 04 00:23:41.507568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.507543    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"]
Nov 04 00:23:41.512615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.512591    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-l7q9b"]
Nov 04 00:23:41.524838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.524801    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"]
Nov 04 00:23:41.529437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:41.529410    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-5rpxh"]
Nov 04 00:23:43.023520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:43.023481    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" path="/var/lib/kubelet/pods/9f45a4f9-10f2-43f3-bda6-492500f39719/volumes"
Nov 04 00:23:43.024123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:43.024083    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" path="/var/lib/kubelet/pods/9f62472d-3985-44e3-ade7-a2a87d28ffcb/volumes"
Nov 04 00:23:46.032025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:46.031978    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"]
Nov 04 00:23:46.036846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:46.036814    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04d51c0-hn8qd"]
Nov 04 00:23:47.024541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:47.024503    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84fbd7a-913f-4f18-a539-d7ab7127c5b5" path="/var/lib/kubelet/pods/f84fbd7a-913f-4f18-a539-d7ab7127c5b5/volumes"
Nov 04 00:23:55.890398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:55.890359    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:23:55.904181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:23:55.904149    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:23:58.026369 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:23:58.026319    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:24:00.488660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:00.488624    5074 scope.go:122] "RemoveContainer" containerID="4a95a0ba596ba1dc8f142f639196159fe3c846ac93686e2011b932026ffe5522"
Nov 04 00:24:00.501960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:00.501933    5074 scope.go:122] "RemoveContainer" containerID="6c912e5e3cfa25a0a44120ef751dad3fcfe88d5bcc622b9186c78081b8aa6ef8"
Nov 04 00:24:08.031906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:08.031866    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"]
Nov 04 00:24:08.043877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:08.043840    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc3d0f51-lktjq"]
Nov 04 00:24:09.023939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:09.023906    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf" path="/var/lib/kubelet/pods/9ee03975-a1f9-4ebc-b64f-2dc45e16e1bf/volumes"
Nov 04 00:24:19.079840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:19.079807    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"]
Nov 04 00:24:19.080246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:19.079850    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b303246-gs5tr"]
Nov 04 00:24:21.022572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:21.022532    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0d7954-5862-4a8f-a614-5b1e215e0f07" path="/var/lib/kubelet/pods/4e0d7954-5862-4a8f-a614-5b1e215e0f07/volumes"
Nov 04 00:24:23.040011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.039979    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"]
Nov 04 00:24:23.041010 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.040988    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy"
Nov 04 00:24:23.041078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041011    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy"
Nov 04 00:24:23.041078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041028    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ba7e4d27-0ada-44d6-989a-098a33b5a4f4" containerName="revoke-keys"
Nov 04 00:24:23.041078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041035    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7e4d27-0ada-44d6-989a-098a33b5a4f4" containerName="revoke-keys"
Nov 04 00:24:23.041078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041052    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" containerName="payload-pre-processing"
Nov 04 00:24:23.041078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041058    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" containerName="payload-pre-processing"
Nov 04 00:24:23.041078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041078    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" containerName="payload-processing"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041086    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" containerName="payload-processing"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041097    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0f98430b-3385-4f3a-8d70-807e6b5b0825" containerName="main"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041102    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f98430b-3385-4f3a-8d70-807e6b5b0825" containerName="main"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041127    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" containerName="payload-processing"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041132    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" containerName="payload-processing"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041144    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041148    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041160    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerName="payload-pre-processing"
Nov 04 00:24:23.041278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041165    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerName="payload-pre-processing"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041305    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7e4d27-0ada-44d6-989a-098a33b5a4f4" containerName="revoke-keys"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041318    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be8c401-cee0-4195-b71e-c76b29f03954" containerName="payload-pre-processing"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041326    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1138f268-e37d-40a9-8214-9b7af5714a37" containerName="payload-processing"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041334    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f98430b-3385-4f3a-8d70-807e6b5b0825" containerName="main"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041343    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a897f01-7e97-40d6-9e16-70deeede895f" containerName="payload-processing"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041349    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f62472d-3985-44e3-ade7-a2a87d28ffcb" containerName="istio-proxy"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041356    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0c9be1-08ce-4e46-bc72-1605dce70b7d" containerName="payload-pre-processing"
Nov 04 00:24:23.041562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.041364    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f45a4f9-10f2-43f3-bda6-492500f39719" containerName="istio-proxy"
Nov 04 00:24:23.048693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.048651    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.053133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.053094    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:24:23.054226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.054198    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"]
Nov 04 00:24:23.055033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.054994    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:24:23.224477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.224430    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.224667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.224505    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xvd\" (UniqueName: \"kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.224813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.224777    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.326384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.326291    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.326549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.326421    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.326549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.326476    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xvd\" (UniqueName: \"kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.327194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.327166    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.328903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.328877    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.339597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.339570    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xvd\" (UniqueName: \"kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd\") pod \"maas-api-revoke-sub-5b4a1c9f-dhzsw\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.377415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.377380    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:23.669768 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:24:23.669703    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9077fc41_c392_456d_960c_07bed701a219.slice/crio-0cdc1358207ff2b5ab793f0e25c8cf7bfbf4d7f69b3b596b8b9e6b06f1942711 WatchSource:0}: Error finding container 0cdc1358207ff2b5ab793f0e25c8cf7bfbf4d7f69b3b596b8b9e6b06f1942711: Status 404 returned error can't find the container with id 0cdc1358207ff2b5ab793f0e25c8cf7bfbf4d7f69b3b596b8b9e6b06f1942711
Nov 04 00:24:23.669999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.669974    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"]
Nov 04 00:24:23.941913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:23.941871    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw" event={"ID":"9077fc41-c392-456d-960c-07bed701a219","Type":"ContainerStarted","Data":"0cdc1358207ff2b5ab793f0e25c8cf7bfbf4d7f69b3b596b8b9e6b06f1942711"}
Nov 04 00:24:24.654807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.654683    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"]
Nov 04 00:24:24.660237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.660208    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.667016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.666981    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"]
Nov 04 00:24:24.753941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.753904    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcm4l\" (UniqueName: \"kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.754128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.754043    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.754175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.754133    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.855442 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.855407    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm4l\" (UniqueName: \"kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.855597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.855548    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.855647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.855599    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.856312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.856281    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.858181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.858154    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.868018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.867996    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcm4l\" (UniqueName: \"kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l\") pod \"maas-api-revoke-sub-91e58553-m4g5h\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:24.958185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.958112    5074 generic.go:356] "Generic (PLEG): container finished" podID="9077fc41-c392-456d-960c-07bed701a219" containerID="c5183a470cd0fde58034653bd3bb8b9c1749204c5b8ad3938d425a685e248a2d" exitCode=0
Nov 04 00:24:24.958401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.958320    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw" event={"ID":"9077fc41-c392-456d-960c-07bed701a219","Type":"ContainerDied","Data":"c5183a470cd0fde58034653bd3bb8b9c1749204c5b8ad3938d425a685e248a2d"}
Nov 04 00:24:24.980203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:24.980170    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:25.282952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:25.282920    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"]
Nov 04 00:24:25.283103 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:24:25.283009    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cb039f_65fb_49f8_b51e_18591725dcfe.slice/crio-e8a788a7c5c86b51a40bf47cec7a68640450c211c102e1e9353b8f86fa3a2dc8 WatchSource:0}: Error finding container e8a788a7c5c86b51a40bf47cec7a68640450c211c102e1e9353b8f86fa3a2dc8: Status 404 returned error can't find the container with id e8a788a7c5c86b51a40bf47cec7a68640450c211c102e1e9353b8f86fa3a2dc8
Nov 04 00:24:25.970426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:25.970321    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h" event={"ID":"54cb039f-65fb-49f8-b51e-18591725dcfe","Type":"ContainerStarted","Data":"a851193a6bde8a6c05d005515b412b8ec036d82ad02ca4fca8e056952f9c5cfd"}
Nov 04 00:24:25.970426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:25.970390    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h" event={"ID":"54cb039f-65fb-49f8-b51e-18591725dcfe","Type":"ContainerStarted","Data":"e8a788a7c5c86b51a40bf47cec7a68640450c211c102e1e9353b8f86fa3a2dc8"}
Nov 04 00:24:25.993483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:25.993429    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h" podStartSLOduration=1.9934106809999999 podStartE2EDuration="1.993410681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:24:25.989298854 +0000 UTC m=+3331.088899010" watchObservedRunningTime="2026-11-04 00:24:25.993410681 +0000 UTC m=+3331.093010825"
Nov 04 00:24:26.218873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.218837    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:26.276787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.276677    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca\") pod \"9077fc41-c392-456d-960c-07bed701a219\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") "
Nov 04 00:24:26.276972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.276868    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd\" (UniqueName: \"kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd\") pod \"9077fc41-c392-456d-960c-07bed701a219\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") "
Nov 04 00:24:26.277098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.277056    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca" pod "9077fc41-c392-456d-960c-07bed701a219" (UID: "9077fc41-c392-456d-960c-07bed701a219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:24:26.277098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.277076    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca\") pod \"9077fc41-c392-456d-960c-07bed701a219\" (UID: \"9077fc41-c392-456d-960c-07bed701a219\") "
Nov 04 00:24:26.277765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.277724    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9077fc41-c392-456d-960c-07bed701a219-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:26.279106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.279084    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca" pod "9077fc41-c392-456d-960c-07bed701a219" (UID: "9077fc41-c392-456d-960c-07bed701a219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:24:26.279238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.279221    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd" pod "9077fc41-c392-456d-960c-07bed701a219" (UID: "9077fc41-c392-456d-960c-07bed701a219"). InnerVolumeSpecName "kube-api-access-f2xvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:26.378689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.378650    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xvd\" (UniqueName: \"kubernetes.io/projected/9077fc41-c392-456d-960c-07bed701a219-kube-api-access-f2xvd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:26.378689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.378685    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9077fc41-c392-456d-960c-07bed701a219-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:26.981712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.981680    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"
Nov 04 00:24:26.981712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.981693    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw" event={"ID":"9077fc41-c392-456d-960c-07bed701a219","Type":"ContainerDied","Data":"0cdc1358207ff2b5ab793f0e25c8cf7bfbf4d7f69b3b596b8b9e6b06f1942711"}
Nov 04 00:24:26.982253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:26.981761    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cdc1358207ff2b5ab793f0e25c8cf7bfbf4d7f69b3b596b8b9e6b06f1942711"
Nov 04 00:24:27.016484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:27.009126    5074 generic.go:356] "Generic (PLEG): container finished" podID="54cb039f-65fb-49f8-b51e-18591725dcfe" containerID="a851193a6bde8a6c05d005515b412b8ec036d82ad02ca4fca8e056952f9c5cfd" exitCode=0
Nov 04 00:24:27.016484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:27.009317    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h" event={"ID":"54cb039f-65fb-49f8-b51e-18591725dcfe","Type":"ContainerDied","Data":"a851193a6bde8a6c05d005515b412b8ec036d82ad02ca4fca8e056952f9c5cfd"}
Nov 04 00:24:27.167094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:27.167051    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"]
Nov 04 00:24:27.167312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:27.167243    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy" containerID="cri-o://f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab" gracePeriod=30
Nov 04 00:24:28.035963 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:24:28.035879    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:24:28.214352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.214321    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:28.343475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.343360    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l\" (UniqueName: \"kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l\") pod \"54cb039f-65fb-49f8-b51e-18591725dcfe\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") "
Nov 04 00:24:28.343800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.343728    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca\") pod \"54cb039f-65fb-49f8-b51e-18591725dcfe\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") "
Nov 04 00:24:28.343939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.343869    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca\") pod \"54cb039f-65fb-49f8-b51e-18591725dcfe\" (UID: \"54cb039f-65fb-49f8-b51e-18591725dcfe\") "
Nov 04 00:24:28.344605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.344564    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca" pod "54cb039f-65fb-49f8-b51e-18591725dcfe" (UID: "54cb039f-65fb-49f8-b51e-18591725dcfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:24:28.346081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.346052    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l" pod "54cb039f-65fb-49f8-b51e-18591725dcfe" (UID: "54cb039f-65fb-49f8-b51e-18591725dcfe"). InnerVolumeSpecName "kube-api-access-gcm4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:28.346199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.346127    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca" pod "54cb039f-65fb-49f8-b51e-18591725dcfe" (UID: "54cb039f-65fb-49f8-b51e-18591725dcfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:24:28.446079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.446026    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcm4l\" (UniqueName: \"kubernetes.io/projected/54cb039f-65fb-49f8-b51e-18591725dcfe-kube-api-access-gcm4l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:28.446079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.446065    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:28.446079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:28.446080    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb039f-65fb-49f8-b51e-18591725dcfe-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:29.040189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:29.033955    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h" event={"ID":"54cb039f-65fb-49f8-b51e-18591725dcfe","Type":"ContainerDied","Data":"e8a788a7c5c86b51a40bf47cec7a68640450c211c102e1e9353b8f86fa3a2dc8"}
Nov 04 00:24:29.040189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:29.033980    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"
Nov 04 00:24:29.040189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:29.034022    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8a788a7c5c86b51a40bf47cec7a68640450c211c102e1e9353b8f86fa3a2dc8"
Nov 04 00:24:32.428891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.428840    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" podUID="14f459d1824c1ea630aa90a99306c647" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:24:32.526235 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.526202    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:24:32.540384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540342    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540396    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540428    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540466    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540522    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540587    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540645    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540731    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.540840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.540820    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw\") pod \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\" (UID: \"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520\") "
Nov 04 00:24:32.543120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.543092    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.545787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.545725    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "kube-api-access-mq4fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:32.546002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.545982    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.546322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.546302    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:24:32.546562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.546527    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.546687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.546660    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.547063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.547035    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:24:32.549905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.549877    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.550062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.550039    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token" pod "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" (UID: "7fcc76b8-d66d-4fc7-97f7-a8842c5f8520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:32.647151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647114    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq4fw\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-kube-api-access-mq4fw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.647364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647347    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.647485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647472    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.647588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647574    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.647684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647672    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.647787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647773    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.647894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647881    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.648004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.647988    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.648094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:32.648083    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:33.076704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.076672    5074 generic.go:356] "Generic (PLEG): container finished" podID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerID="f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab" exitCode=0
Nov 04 00:24:33.076918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.076716    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"
Nov 04 00:24:33.076918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.076762    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" event={"ID":"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520","Type":"ContainerDied","Data":"f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab"}
Nov 04 00:24:33.076918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.076821    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9" event={"ID":"7fcc76b8-d66d-4fc7-97f7-a8842c5f8520","Type":"ContainerDied","Data":"71520ebe61a3f5896542ee0e9248d9ad5ffc298a00f81cab2240123dd4a5e91a"}
Nov 04 00:24:33.076918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.076846    5074 scope.go:122] "RemoveContainer" containerID="f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab"
Nov 04 00:24:33.089283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.089266    5074 scope.go:122] "RemoveContainer" containerID="f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab"
Nov 04 00:24:33.089609 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:24:33.089572    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab\": container with ID starting with f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab not found: ID does not exist" containerID="f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab"
Nov 04 00:24:33.089719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.089616    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab"} err="failed to get container status \"f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab\": rpc error: code = NotFound desc = could not find container \"f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab\": container with ID starting with f8a7532974098cba1ca8c2f7bc420a1ccfb592cf5861030ab2cd4bcb6c8553ab not found: ID does not exist"
Nov 04 00:24:33.103067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.103044    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"]
Nov 04 00:24:33.107666 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:33.107639    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-gf4f9"]
Nov 04 00:24:34.223586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.222753    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"]
Nov 04 00:24:34.224437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.224412    5074 cpu_manager.go:414] "containerMap: removing container" podUID="54cb039f-65fb-49f8-b51e-18591725dcfe" containerName="revoke-keys"
Nov 04 00:24:34.224624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.224610    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cb039f-65fb-49f8-b51e-18591725dcfe" containerName="revoke-keys"
Nov 04 00:24:34.224812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.224799    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy"
Nov 04 00:24:34.224928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.224908    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy"
Nov 04 00:24:34.225042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.224936    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9077fc41-c392-456d-960c-07bed701a219" containerName="revoke-keys"
Nov 04 00:24:34.225042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.224944    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9077fc41-c392-456d-960c-07bed701a219" containerName="revoke-keys"
Nov 04 00:24:34.225201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.225185    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" containerName="istio-proxy"
Nov 04 00:24:34.225418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.225398    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9077fc41-c392-456d-960c-07bed701a219" containerName="revoke-keys"
Nov 04 00:24:34.225516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.225440    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cb039f-65fb-49f8-b51e-18591725dcfe" containerName="revoke-keys"
Nov 04 00:24:34.232969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.232947    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.242918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.242889    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:24:34.243212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.243008    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:24:34.251977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.251581    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"]
Nov 04 00:24:34.275051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.275019    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.275267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.275237    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbh8x\" (UniqueName: \"kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.275473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.275452    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.377105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.377057    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.377306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.377179    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.377306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.377294    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbh8x\" (UniqueName: \"kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.378082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.378053    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.379851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.379825    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.390186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.390165    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbh8x\" (UniqueName: \"kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x\") pod \"maas-api-revoke-sub-f5e20d50-2gddm\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.560444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.560363    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:34.853978 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:24:34.853945    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85a8d8f_62e4_42d3_a495_ae8f7e04625a.slice/crio-ed39ab6d314ce0a0fc87c2c2d14b7d820620eee27f6c705d0f7c5dba6f512a44 WatchSource:0}: Error finding container ed39ab6d314ce0a0fc87c2c2d14b7d820620eee27f6c705d0f7c5dba6f512a44: Status 404 returned error can't find the container with id ed39ab6d314ce0a0fc87c2c2d14b7d820620eee27f6c705d0f7c5dba6f512a44
Nov 04 00:24:34.854188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:34.854166    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"]
Nov 04 00:24:35.023958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:35.023925    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcc76b8-d66d-4fc7-97f7-a8842c5f8520" path="/var/lib/kubelet/pods/7fcc76b8-d66d-4fc7-97f7-a8842c5f8520/volumes"
Nov 04 00:24:35.096484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:35.096451    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm" event={"ID":"e85a8d8f-62e4-42d3-a495-ae8f7e04625a","Type":"ContainerStarted","Data":"ed39ab6d314ce0a0fc87c2c2d14b7d820620eee27f6c705d0f7c5dba6f512a44"}
Nov 04 00:24:36.107877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:36.107838    5074 generic.go:356] "Generic (PLEG): container finished" podID="e85a8d8f-62e4-42d3-a495-ae8f7e04625a" containerID="ceda4767db20d01fb6a50e8423f038faf2d5be1868ddb3a20be92d488b236911" exitCode=0
Nov 04 00:24:36.108289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:36.107922    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm" event={"ID":"e85a8d8f-62e4-42d3-a495-ae8f7e04625a","Type":"ContainerDied","Data":"ceda4767db20d01fb6a50e8423f038faf2d5be1868ddb3a20be92d488b236911"}
Nov 04 00:24:37.304355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.304318    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:37.333143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.333110    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x\" (UniqueName: \"kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x\") pod \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") "
Nov 04 00:24:37.333289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.333172    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca\") pod \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") "
Nov 04 00:24:37.333349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.333309    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca\") pod \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\" (UID: \"e85a8d8f-62e4-42d3-a495-ae8f7e04625a\") "
Nov 04 00:24:37.333853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.333820    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca" pod "e85a8d8f-62e4-42d3-a495-ae8f7e04625a" (UID: "e85a8d8f-62e4-42d3-a495-ae8f7e04625a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:24:37.334312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.334288    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:37.336437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.336402    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x" pod "e85a8d8f-62e4-42d3-a495-ae8f7e04625a" (UID: "e85a8d8f-62e4-42d3-a495-ae8f7e04625a"). InnerVolumeSpecName "kube-api-access-lbh8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:37.336869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.336846    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca" pod "e85a8d8f-62e4-42d3-a495-ae8f7e04625a" (UID: "e85a8d8f-62e4-42d3-a495-ae8f7e04625a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:24:37.436483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.436452    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbh8x\" (UniqueName: \"kubernetes.io/projected/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-kube-api-access-lbh8x\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:37.436483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:37.436484    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85a8d8f-62e4-42d3-a495-ae8f7e04625a-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:38.133491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:38.133450    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm" event={"ID":"e85a8d8f-62e4-42d3-a495-ae8f7e04625a","Type":"ContainerDied","Data":"ed39ab6d314ce0a0fc87c2c2d14b7d820620eee27f6c705d0f7c5dba6f512a44"}
Nov 04 00:24:38.133491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:38.133475    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"
Nov 04 00:24:38.133758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:38.133508    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed39ab6d314ce0a0fc87c2c2d14b7d820620eee27f6c705d0f7c5dba6f512a44"
Nov 04 00:24:39.243393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.243353    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"]
Nov 04 00:24:39.244704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.244676    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e85a8d8f-62e4-42d3-a495-ae8f7e04625a" containerName="revoke-keys"
Nov 04 00:24:39.244704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.244702    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85a8d8f-62e4-42d3-a495-ae8f7e04625a" containerName="revoke-keys"
Nov 04 00:24:39.245018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.245000    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85a8d8f-62e4-42d3-a495-ae8f7e04625a" containerName="revoke-keys"
Nov 04 00:24:39.249474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.249450    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.253893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.253868    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:24:39.255125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.255102    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"]
Nov 04 00:24:39.255861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.255840    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:24:39.348525 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.348491    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.348764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.348693    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc46\" (UniqueName: \"kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.348889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.348869    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.450524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.450473    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.450729 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.450671    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.450877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.450860    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc46\" (UniqueName: \"kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.451255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.451227    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.453183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.453164    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.463879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.463857    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc46\" (UniqueName: \"kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46\") pod \"maas-api-revoke-sub-33619f16-5cx82\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.568980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.568892    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:39.864992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:39.864953    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"]
Nov 04 00:24:39.865429 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:24:39.865406    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396e8e57_5200_4abf_a4f5_23dc5152f440.slice/crio-8eeb1b839ebfcaeadcd60e3eef857c7b704b8e00328eb4137add744f19e7266b WatchSource:0}: Error finding container 8eeb1b839ebfcaeadcd60e3eef857c7b704b8e00328eb4137add744f19e7266b: Status 404 returned error can't find the container with id 8eeb1b839ebfcaeadcd60e3eef857c7b704b8e00328eb4137add744f19e7266b
Nov 04 00:24:40.157108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:40.157052    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82" event={"ID":"396e8e57-5200-4abf-a4f5-23dc5152f440","Type":"ContainerStarted","Data":"8eeb1b839ebfcaeadcd60e3eef857c7b704b8e00328eb4137add744f19e7266b"}
Nov 04 00:24:41.167195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:41.167117    5074 generic.go:356] "Generic (PLEG): container finished" podID="396e8e57-5200-4abf-a4f5-23dc5152f440" containerID="67206a1e5103ecf71faa373c95b1858193fb485b52e40b2de40ff6624cc312f1" exitCode=0
Nov 04 00:24:41.167569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:41.167203    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82" event={"ID":"396e8e57-5200-4abf-a4f5-23dc5152f440","Type":"ContainerDied","Data":"67206a1e5103ecf71faa373c95b1858193fb485b52e40b2de40ff6624cc312f1"}
Nov 04 00:24:42.390290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.390263    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:42.495976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.495942    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca\") pod \"396e8e57-5200-4abf-a4f5-23dc5152f440\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") "
Nov 04 00:24:42.496177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.496163    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca\") pod \"396e8e57-5200-4abf-a4f5-23dc5152f440\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") "
Nov 04 00:24:42.496226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.496193    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46\" (UniqueName: \"kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46\") pod \"396e8e57-5200-4abf-a4f5-23dc5152f440\" (UID: \"396e8e57-5200-4abf-a4f5-23dc5152f440\") "
Nov 04 00:24:42.496579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.496551    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca" pod "396e8e57-5200-4abf-a4f5-23dc5152f440" (UID: "396e8e57-5200-4abf-a4f5-23dc5152f440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:24:42.496881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.496863    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:42.498363 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.498327    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca" pod "396e8e57-5200-4abf-a4f5-23dc5152f440" (UID: "396e8e57-5200-4abf-a4f5-23dc5152f440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:24:42.498483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.498442    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46" pod "396e8e57-5200-4abf-a4f5-23dc5152f440" (UID: "396e8e57-5200-4abf-a4f5-23dc5152f440"). InnerVolumeSpecName "kube-api-access-2rc46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:42.598933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.598887    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396e8e57-5200-4abf-a4f5-23dc5152f440-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:42.598933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:42.598929    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc46\" (UniqueName: \"kubernetes.io/projected/396e8e57-5200-4abf-a4f5-23dc5152f440-kube-api-access-2rc46\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:43.185638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:43.185608    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"
Nov 04 00:24:43.185834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:43.185606    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82" event={"ID":"396e8e57-5200-4abf-a4f5-23dc5152f440","Type":"ContainerDied","Data":"8eeb1b839ebfcaeadcd60e3eef857c7b704b8e00328eb4137add744f19e7266b"}
Nov 04 00:24:43.185834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:24:43.185752    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eeb1b839ebfcaeadcd60e3eef857c7b704b8e00328eb4137add744f19e7266b"
Nov 04 00:24:58.033960 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:24:58.033917    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:25:00.580249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:00.580212    5074 scope.go:122] "RemoveContainer" containerID="4fb6179edd73c92bb498c93a58a95986445b235890fcb5d5344a5d37db693d06"
Nov 04 00:25:00.592449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:00.592428    5074 scope.go:122] "RemoveContainer" containerID="d1504a31a04c8de25a042b24e6887f6114177bb817e5fe54b9613a8483ec6c4b"
Nov 04 00:25:03.768001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:03.767964    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"]
Nov 04 00:25:03.869601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:03.869564    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"]
Nov 04 00:25:04.974416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:04.974379    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"]
Nov 04 00:25:04.990807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:04.990775    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"]
Nov 04 00:25:08.768832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:08.768785    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerName="payload-processing" containerID="cri-o://e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09" gracePeriod=25
Nov 04 00:25:08.870693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:08.870650    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerName="payload-pre-processing" containerID="cri-o://a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea" gracePeriod=25
Nov 04 00:25:09.975254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:09.975210    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7" podUID="9c3d0fed-c06e-4eab-bcb1-9840298d2404" containerName="payload-processing" containerID="cri-o://4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8" gracePeriod=25
Nov 04 00:25:09.991280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:09.991236    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerName="payload-pre-processing" containerID="cri-o://4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72" gracePeriod=25
Nov 04 00:25:10.281974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.281943    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:25:10.393724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.393681    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume\") pod \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") "
Nov 04 00:25:10.393989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.393970    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst\" (UniqueName: \"kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst\") pod \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\" (UID: \"9c3d0fed-c06e-4eab-bcb1-9840298d2404\") "
Nov 04 00:25:10.394156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.394125    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume" pod "9c3d0fed-c06e-4eab-bcb1-9840298d2404" (UID: "9c3d0fed-c06e-4eab-bcb1-9840298d2404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:10.394546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.394526    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d0fed-c06e-4eab-bcb1-9840298d2404-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:10.396320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.396299    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst" pod "9c3d0fed-c06e-4eab-bcb1-9840298d2404" (UID: "9c3d0fed-c06e-4eab-bcb1-9840298d2404"). InnerVolumeSpecName "kube-api-access-gzqst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:10.496653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.496558    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqst\" (UniqueName: \"kubernetes.io/projected/9c3d0fed-c06e-4eab-bcb1-9840298d2404-kube-api-access-gzqst\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:10.508873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.508797    5074 generic.go:356] "Generic (PLEG): container finished" podID="9c3d0fed-c06e-4eab-bcb1-9840298d2404" containerID="4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8" exitCode=0
Nov 04 00:25:10.509050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.508965    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"
Nov 04 00:25:10.509050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.508990    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7" event={"ID":"9c3d0fed-c06e-4eab-bcb1-9840298d2404","Type":"ContainerDied","Data":"4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8"}
Nov 04 00:25:10.509193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.509095    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7" event={"ID":"9c3d0fed-c06e-4eab-bcb1-9840298d2404","Type":"ContainerDied","Data":"20b3eba2050dcaed489a0f80191e7a6feebad52d1fc09604e4f31896c38227d6"}
Nov 04 00:25:10.509193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.509130    5074 scope.go:122] "RemoveContainer" containerID="4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8"
Nov 04 00:25:10.524805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.524731    5074 scope.go:122] "RemoveContainer" containerID="4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8"
Nov 04 00:25:10.525099 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:10.525075    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8\": container with ID starting with 4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8 not found: ID does not exist" containerID="4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8"
Nov 04 00:25:10.525174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.525114    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8"} err="failed to get container status \"4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8\": rpc error: code = NotFound desc = could not find container \"4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8\": container with ID starting with 4b2cc8e55ffde4311cfecf70a1fe654849ea0f1376cd869783de23165d2bcac8 not found: ID does not exist"
Nov 04 00:25:10.537279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.537251    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"]
Nov 04 00:25:10.541722 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:10.541694    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-954cd9-7478bd97bd-pkph7"]
Nov 04 00:25:11.022655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:11.022620    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3d0fed-c06e-4eab-bcb1-9840298d2404" path="/var/lib/kubelet/pods/9c3d0fed-c06e-4eab-bcb1-9840298d2404/volumes"
Nov 04 00:25:11.052847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:11.052797    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:11.053060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:11.053023    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:12.201505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.201445    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:12.697773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.697716    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"]
Nov 04 00:25:12.698902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.698874    5074 cpu_manager.go:414] "containerMap: removing container" podUID="396e8e57-5200-4abf-a4f5-23dc5152f440" containerName="revoke-keys"
Nov 04 00:25:12.698902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.698899    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396e8e57-5200-4abf-a4f5-23dc5152f440" containerName="revoke-keys"
Nov 04 00:25:12.699051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.698921    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9c3d0fed-c06e-4eab-bcb1-9840298d2404" containerName="payload-processing"
Nov 04 00:25:12.699051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.698927    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3d0fed-c06e-4eab-bcb1-9840298d2404" containerName="payload-processing"
Nov 04 00:25:12.699125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.699068    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="396e8e57-5200-4abf-a4f5-23dc5152f440" containerName="revoke-keys"
Nov 04 00:25:12.699125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.699087    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3d0fed-c06e-4eab-bcb1-9840298d2404" containerName="payload-processing"
Nov 04 00:25:12.705202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.705180    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.722917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.722888    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"]
Nov 04 00:25:12.830883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.830847    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpfk\" (UniqueName: \"kubernetes.io/projected/060dc557-cd0d-437a-b84e-6e52bb4e79a1-kube-api-access-dkpfk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.830902    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831023    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831056    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831080    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831189    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831274    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831314    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.831370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.831347    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.932806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.932768    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.932979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.932874    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.932979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.932916    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.932979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.932950    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933018    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpfk\" (UniqueName: \"kubernetes.io/projected/060dc557-cd0d-437a-b84e-6e52bb4e79a1-kube-api-access-dkpfk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933041    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933095    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933132    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933161    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933343    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933445    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933497 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933471    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933536    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.933604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.933572    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.935677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.935648    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.936137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.936119    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.945971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.945949    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpfk\" (UniqueName: \"kubernetes.io/projected/060dc557-cd0d-437a-b84e-6e52bb4e79a1-kube-api-access-dkpfk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:12.946091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:12.945970    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060dc557-cd0d-437a-b84e-6e52bb4e79a1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nlkqs\" (UID: \"060dc557-cd0d-437a-b84e-6e52bb4e79a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:13.031875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.031800    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:13.357852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.357821    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"]
Nov 04 00:25:13.358410 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:25:13.358384    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060dc557_cd0d_437a_b84e_6e52bb4e79a1.slice/crio-d955a85c98a71ca0f797597edc68ec55573f15e7a302c1ffbdaab86db10a7f0e WatchSource:0}: Error finding container d955a85c98a71ca0f797597edc68ec55573f15e7a302c1ffbdaab86db10a7f0e: Status 404 returned error can't find the container with id d955a85c98a71ca0f797597edc68ec55573f15e7a302c1ffbdaab86db10a7f0e
Nov 04 00:25:13.362276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.362236    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:25:13.362384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.362322    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:25:13.362429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.362393    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:25:13.541927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.541879    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" event={"ID":"060dc557-cd0d-437a-b84e-6e52bb4e79a1","Type":"ContainerStarted","Data":"d839f27bcc4f2edbe1d20b05a0281c8ff5fa06a65697e8718766ddce047c7926"}
Nov 04 00:25:13.542136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.541957    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" event={"ID":"060dc557-cd0d-437a-b84e-6e52bb4e79a1","Type":"ContainerStarted","Data":"d955a85c98a71ca0f797597edc68ec55573f15e7a302c1ffbdaab86db10a7f0e"}
Nov 04 00:25:13.569916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:13.569852    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podStartSLOduration=1.5698341249999999 podStartE2EDuration="1.569834125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:25:13.568474149 +0000 UTC m=+3378.668074293" watchObservedRunningTime="2026-11-04 00:25:13.569834125 +0000 UTC m=+3378.669434271"
Nov 04 00:25:14.032974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:14.032941    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:14.034684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:14.034655    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:14.034847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:14.034708    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:15.032296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:15.032260    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:15.032840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:15.032322    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:16.032985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:16.032945    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:16.033419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:16.033014    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:17.032241 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:17.032202    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:17.032461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:17.032265    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:18.034763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:18.033959    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:18.034763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:18.034035    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:19.064979 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:19.064942    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:19.074944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:19.074905    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:20.034660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:20.032577    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:20.034660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:20.032662    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:21.032834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:21.032797    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body=
Nov 04 00:25:21.033426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:21.033390    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused"
Nov 04 00:25:21.052361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:21.052324    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:21.052361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:21.052342    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:22.202929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:22.202889    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:22.644136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:22.644056    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:22.644707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:22.644676    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:22.647659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:22.647632    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs"
Nov 04 00:25:28.042812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:28.042767    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"]
Nov 04 00:25:28.044312 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:28.044277    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:25:28.055002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:28.054967    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"]
Nov 04 00:25:29.097732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.097632    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:25:29.194830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.194795    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:25:29.235522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.235486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks\" (UniqueName: \"kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks\") pod \"c77a574c-4f20-42c2-be0e-94c8ce71b384\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") "
Nov 04 00:25:29.235719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.235659    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume\") pod \"c77a574c-4f20-42c2-be0e-94c8ce71b384\" (UID: \"c77a574c-4f20-42c2-be0e-94c8ce71b384\") "
Nov 04 00:25:29.236369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.236122    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume" pod "c77a574c-4f20-42c2-be0e-94c8ce71b384" (UID: "c77a574c-4f20-42c2-be0e-94c8ce71b384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:29.236369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.236285    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c77a574c-4f20-42c2-be0e-94c8ce71b384-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:29.238403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.238377    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks" pod "c77a574c-4f20-42c2-be0e-94c8ce71b384" (UID: "c77a574c-4f20-42c2-be0e-94c8ce71b384"). InnerVolumeSpecName "kube-api-access-srzks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:29.337514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.337402    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj\" (UniqueName: \"kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj\") pod \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") "
Nov 04 00:25:29.337712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.337544    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume\") pod \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\" (UID: \"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7\") "
Nov 04 00:25:29.338061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.338031    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume" pod "0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" (UID: "0dc36f37-efbc-4342-b9ff-3cd5e8d630a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:29.338258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.338236    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzks\" (UniqueName: \"kubernetes.io/projected/c77a574c-4f20-42c2-be0e-94c8ce71b384-kube-api-access-srzks\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:29.338325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.338264    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:29.339772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.339705    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj" pod "0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" (UID: "0dc36f37-efbc-4342-b9ff-3cd5e8d630a7"). InnerVolumeSpecName "kube-api-access-xn9zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:29.440574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.440533    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9zj\" (UniqueName: \"kubernetes.io/projected/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7-kube-api-access-xn9zj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:29.844413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.844372    5074 generic.go:356] "Generic (PLEG): container finished" podID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerID="e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09" exitCode=0
Nov 04 00:25:29.844627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.844434    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"
Nov 04 00:25:29.844627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.844454    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" event={"ID":"c77a574c-4f20-42c2-be0e-94c8ce71b384","Type":"ContainerDied","Data":"e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09"}
Nov 04 00:25:29.844627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.844547    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7" event={"ID":"c77a574c-4f20-42c2-be0e-94c8ce71b384","Type":"ContainerDied","Data":"ad43944c924b91ebbe106c89466dc27bdcbb8a4d267db52afbd3edcce9275731"}
Nov 04 00:25:29.844627 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.844581    5074 scope.go:122] "RemoveContainer" containerID="e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09"
Nov 04 00:25:29.846984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.846959    5074 generic.go:356] "Generic (PLEG): container finished" podID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerID="a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea" exitCode=0
Nov 04 00:25:29.847152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.847026    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" event={"ID":"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7","Type":"ContainerDied","Data":"a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea"}
Nov 04 00:25:29.847152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.847059    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8" event={"ID":"0dc36f37-efbc-4342-b9ff-3cd5e8d630a7","Type":"ContainerDied","Data":"f812cf72bde3fb9ce42aa5d0f2ca53ee76ad537154780b476f6e9fe4d54c1617"}
Nov 04 00:25:29.847152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.847123    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"
Nov 04 00:25:29.858196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.858178    5074 scope.go:122] "RemoveContainer" containerID="e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09"
Nov 04 00:25:29.858481 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:29.858459    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09\": container with ID starting with e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09 not found: ID does not exist" containerID="e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09"
Nov 04 00:25:29.858577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.858487    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09"} err="failed to get container status \"e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09\": rpc error: code = NotFound desc = could not find container \"e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09\": container with ID starting with e4c0a06ef6418110257ed2cb43e8b3353003c3db2d9bf836bad7ed6b7fc65b09 not found: ID does not exist"
Nov 04 00:25:29.858577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.858504    5074 scope.go:122] "RemoveContainer" containerID="a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea"
Nov 04 00:25:29.869145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.869124    5074 scope.go:122] "RemoveContainer" containerID="a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea"
Nov 04 00:25:29.869401 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:29.869378    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea\": container with ID starting with a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea not found: ID does not exist" containerID="a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea"
Nov 04 00:25:29.869499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.869406    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea"} err="failed to get container status \"a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea\": rpc error: code = NotFound desc = could not find container \"a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea\": container with ID starting with a8b9806097fa3b612987938c790f70d8b02b2b52f716601a579c42df4890f1ea not found: ID does not exist"
Nov 04 00:25:29.876023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.875998    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"]
Nov 04 00:25:29.880582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.880557    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a6f7a-69774dc54d-vzhm7"]
Nov 04 00:25:29.891385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.891365    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"]
Nov 04 00:25:29.895623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:29.895603    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a6f7a-687674cc6c-sc9p8"]
Nov 04 00:25:30.689938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.689908    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:25:30.788196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.788157    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw\" (UniqueName: \"kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw\") pod \"40a40e38-20a8-4ae3-81e6-480811a4049b\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") "
Nov 04 00:25:30.788395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.788327    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume\") pod \"40a40e38-20a8-4ae3-81e6-480811a4049b\" (UID: \"40a40e38-20a8-4ae3-81e6-480811a4049b\") "
Nov 04 00:25:30.789267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.789241    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume" pod "40a40e38-20a8-4ae3-81e6-480811a4049b" (UID: "40a40e38-20a8-4ae3-81e6-480811a4049b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:30.790937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.790908    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw" pod "40a40e38-20a8-4ae3-81e6-480811a4049b" (UID: "40a40e38-20a8-4ae3-81e6-480811a4049b"). InnerVolumeSpecName "kube-api-access-n78fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:30.858197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.858170    5074 generic.go:356] "Generic (PLEG): container finished" podID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerID="4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72" exitCode=0
Nov 04 00:25:30.858362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.858219    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"
Nov 04 00:25:30.858362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.858252    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" event={"ID":"40a40e38-20a8-4ae3-81e6-480811a4049b","Type":"ContainerDied","Data":"4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72"}
Nov 04 00:25:30.858362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.858307    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn" event={"ID":"40a40e38-20a8-4ae3-81e6-480811a4049b","Type":"ContainerDied","Data":"e70f0bb838e469853a92ad0c53f7f2f1539413d39626af9953245f559025c810"}
Nov 04 00:25:30.858362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.858336    5074 scope.go:122] "RemoveContainer" containerID="4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72"
Nov 04 00:25:30.871211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.871191    5074 scope.go:122] "RemoveContainer" containerID="4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72"
Nov 04 00:25:30.871450 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:30.871431    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72\": container with ID starting with 4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72 not found: ID does not exist" containerID="4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72"
Nov 04 00:25:30.871514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.871459    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72"} err="failed to get container status \"4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72\": rpc error: code = NotFound desc = could not find container \"4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72\": container with ID starting with 4a8d023bae5ee6521c750c313efaf4ada83d66c808e5f99b98931c16ca894c72 not found: ID does not exist"
Nov 04 00:25:30.888333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.888307    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"]
Nov 04 00:25:30.890484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.890453    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78fw\" (UniqueName: \"kubernetes.io/projected/40a40e38-20a8-4ae3-81e6-480811a4049b-kube-api-access-n78fw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:30.890484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.890478    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a40e38-20a8-4ae3-81e6-480811a4049b-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:30.895101 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:30.895079    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-954cd9-d4564569f-84wpn"]
Nov 04 00:25:31.020838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:31.020799    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" path="/var/lib/kubelet/pods/0dc36f37-efbc-4342-b9ff-3cd5e8d630a7/volumes"
Nov 04 00:25:31.021306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:31.021289    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" path="/var/lib/kubelet/pods/40a40e38-20a8-4ae3-81e6-480811a4049b/volumes"
Nov 04 00:25:31.021685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:31.021670    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" path="/var/lib/kubelet/pods/c77a574c-4f20-42c2-be0e-94c8ce71b384/volumes"
Nov 04 00:25:33.043459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.043411    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5" podUID="e560ec02-fac5-45d3-bfd7-c9c6291bf776" containerName="payload-processing" containerID="cri-o://8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8" gracePeriod=25
Nov 04 00:25:33.056055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.056012    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerName="payload-pre-processing" containerID="cri-o://8379205087a9ddb46881de60cef4c12f12cf9eb5d3d4605c23067123d2a30bdf" gracePeriod=25
Nov 04 00:25:33.356531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.356500    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:25:33.456332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.456297    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume\") pod \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") "
Nov 04 00:25:33.456523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.456362    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9\" (UniqueName: \"kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9\") pod \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\" (UID: \"e560ec02-fac5-45d3-bfd7-c9c6291bf776\") "
Nov 04 00:25:33.456670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.456647    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume" pod "e560ec02-fac5-45d3-bfd7-c9c6291bf776" (UID: "e560ec02-fac5-45d3-bfd7-c9c6291bf776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:33.456827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.456812    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e560ec02-fac5-45d3-bfd7-c9c6291bf776-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:33.458918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.458890    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9" pod "e560ec02-fac5-45d3-bfd7-c9c6291bf776" (UID: "e560ec02-fac5-45d3-bfd7-c9c6291bf776"). InnerVolumeSpecName "kube-api-access-5p2t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:33.558548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.558502    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2t9\" (UniqueName: \"kubernetes.io/projected/e560ec02-fac5-45d3-bfd7-c9c6291bf776-kube-api-access-5p2t9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:33.887185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.887150    5074 generic.go:356] "Generic (PLEG): container finished" podID="e560ec02-fac5-45d3-bfd7-c9c6291bf776" containerID="8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8" exitCode=0
Nov 04 00:25:33.887370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.887195    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"
Nov 04 00:25:33.887370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.887234    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5" event={"ID":"e560ec02-fac5-45d3-bfd7-c9c6291bf776","Type":"ContainerDied","Data":"8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8"}
Nov 04 00:25:33.887370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.887286    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5" event={"ID":"e560ec02-fac5-45d3-bfd7-c9c6291bf776","Type":"ContainerDied","Data":"e096e121c125ffc839f81c9d54c7fa01a9578adb9e283a95628a84e60f90929b"}
Nov 04 00:25:33.887370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.887321    5074 scope.go:122] "RemoveContainer" containerID="8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8"
Nov 04 00:25:33.946433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.946401    5074 scope.go:122] "RemoveContainer" containerID="8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8"
Nov 04 00:25:33.947200 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:33.947169    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8\": container with ID starting with 8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8 not found: ID does not exist" containerID="8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8"
Nov 04 00:25:33.947337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.947211    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8"} err="failed to get container status \"8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8\": rpc error: code = NotFound desc = could not find container \"8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8\": container with ID starting with 8c1f14bae8c1758054b4f7ba8f1abceb17f110c399373fd9cdbff40a46121ac8 not found: ID does not exist"
Nov 04 00:25:33.949582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.949558    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"]
Nov 04 00:25:33.955829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:33.955800    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7cbc3-67bcbfb584-2m9j5"]
Nov 04 00:25:35.022869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:35.022830    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e560ec02-fac5-45d3-bfd7-c9c6291bf776" path="/var/lib/kubelet/pods/e560ec02-fac5-45d3-bfd7-c9c6291bf776/volumes"
Nov 04 00:25:38.822182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:38.822125    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:48.820870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:48.820823    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:25:53.195038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.195006    5074 generic.go:356] "Generic (PLEG): container finished" podID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerID="8379205087a9ddb46881de60cef4c12f12cf9eb5d3d4605c23067123d2a30bdf" exitCode=0
Nov 04 00:25:53.195387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.195078    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" event={"ID":"03935378-bfbf-4b4e-a95a-13b693cab31c","Type":"ContainerDied","Data":"8379205087a9ddb46881de60cef4c12f12cf9eb5d3d4605c23067123d2a30bdf"}
Nov 04 00:25:53.368959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.368934    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:25:53.463579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.463550    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl\" (UniqueName: \"kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl\") pod \"03935378-bfbf-4b4e-a95a-13b693cab31c\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") "
Nov 04 00:25:53.463778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.463751    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume\") pod \"03935378-bfbf-4b4e-a95a-13b693cab31c\" (UID: \"03935378-bfbf-4b4e-a95a-13b693cab31c\") "
Nov 04 00:25:53.464233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.464203    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume" pod "03935378-bfbf-4b4e-a95a-13b693cab31c" (UID: "03935378-bfbf-4b4e-a95a-13b693cab31c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:53.465766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.465730    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl" pod "03935378-bfbf-4b4e-a95a-13b693cab31c" (UID: "03935378-bfbf-4b4e-a95a-13b693cab31c"). InnerVolumeSpecName "kube-api-access-kqfkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:53.564944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.564911    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03935378-bfbf-4b4e-a95a-13b693cab31c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:53.564944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:53.564941    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfkl\" (UniqueName: \"kubernetes.io/projected/03935378-bfbf-4b4e-a95a-13b693cab31c-kube-api-access-kqfkl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.209035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:54.208999    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"
Nov 04 00:25:54.209589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:54.209040    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58" event={"ID":"03935378-bfbf-4b4e-a95a-13b693cab31c","Type":"ContainerDied","Data":"471ee90bfcf4edbd5c911ada75e58ad55156a168af0b47f4ea2d26c71c5641b6"}
Nov 04 00:25:54.209589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:54.209123    5074 scope.go:122] "RemoveContainer" containerID="8379205087a9ddb46881de60cef4c12f12cf9eb5d3d4605c23067123d2a30bdf"
Nov 04 00:25:54.241262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:54.241226    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"]
Nov 04 00:25:54.257932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:54.257142    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7cbc3-587fbd6959-45p58"]
Nov 04 00:25:55.058764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:55.057976    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" path="/var/lib/kubelet/pods/03935378-bfbf-4b4e-a95a-13b693cab31c/volumes"
Nov 04 00:25:58.025292 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:25:58.025251    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:25:59.735219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.735179    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"]
Nov 04 00:25:59.736307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736287    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerName="payload-pre-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736309    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerName="payload-pre-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736322    5074 cpu_manager.go:414] "containerMap: removing container" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerName="payload-pre-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736327    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerName="payload-pre-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736341    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerName="payload-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736348    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerName="payload-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736355    5074 cpu_manager.go:414] "containerMap: removing container" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerName="payload-pre-processing"
Nov 04 00:25:59.736364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736359    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerName="payload-pre-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736400    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e560ec02-fac5-45d3-bfd7-c9c6291bf776" containerName="payload-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736405    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e560ec02-fac5-45d3-bfd7-c9c6291bf776" containerName="payload-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736540    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e560ec02-fac5-45d3-bfd7-c9c6291bf776" containerName="payload-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736552    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc36f37-efbc-4342-b9ff-3cd5e8d630a7" containerName="payload-pre-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736560    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03935378-bfbf-4b4e-a95a-13b693cab31c" containerName="payload-pre-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736567    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77a574c-4f20-42c2-be0e-94c8ce71b384" containerName="payload-processing"
Nov 04 00:25:59.736617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.736573    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a40e38-20a8-4ae3-81e6-480811a4049b" containerName="payload-pre-processing"
Nov 04 00:25:59.741170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.741145    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.753281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.753252    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"]
Nov 04 00:25:59.758884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.756125    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:25:59.758884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.756235    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:25:59.771612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.767041    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgj6\" (UniqueName: \"kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.771612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.767104    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.771612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.767254    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.869244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.869207    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgj6\" (UniqueName: \"kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.869428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.869261    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.869428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.869400    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.870170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.870137    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.871694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.871671    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:25:59.884011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:25:59.883989    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgj6\" (UniqueName: \"kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6\") pod \"maas-api-revoke-sub-927315ed-qrvhs\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:26:00.097618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:00.097532    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:26:00.397135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:00.397104    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"]
Nov 04 00:26:00.646421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:00.646316    5074 scope.go:122] "RemoveContainer" containerID="70800aed2e453cc21f341064debea92d26a09bf083b8c5f317a4dbb08b530b86"
Nov 04 00:26:00.658471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:00.658441    5074 scope.go:122] "RemoveContainer" containerID="abf4e68e0c4f400ca7cda6b333c4a6fc79339f1dfdd265430d4c0e03acd5f46b"
Nov 04 00:26:01.315925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:01.315811    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs" event={"ID":"65935fe5-8739-4fe3-b88b-567944039c03","Type":"ContainerStarted","Data":"8436bcc8be5f82f0e9b569b3d005bee71a163c1f41fba471f01ea19b953d9bc7"}
Nov 04 00:26:01.315925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:01.315883    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs" event={"ID":"65935fe5-8739-4fe3-b88b-567944039c03","Type":"ContainerStarted","Data":"8d4fe8db73258f89fe80da04fb3ae4c62a262359ba7ec0c748aab1cd077d73f9"}
Nov 04 00:26:01.338116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:01.338061    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs" podStartSLOduration=2.338046716 podStartE2EDuration="2.338046716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:26:01.337001884 +0000 UTC m=+3426.436602030" watchObservedRunningTime="2026-11-04 00:26:01.338046716 +0000 UTC m=+3426.437646875"
Nov 04 00:26:02.328016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:02.327976    5074 generic.go:356] "Generic (PLEG): container finished" podID="65935fe5-8739-4fe3-b88b-567944039c03" containerID="8436bcc8be5f82f0e9b569b3d005bee71a163c1f41fba471f01ea19b953d9bc7" exitCode=0
Nov 04 00:26:02.328433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:02.328053    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs" event={"ID":"65935fe5-8739-4fe3-b88b-567944039c03","Type":"ContainerDied","Data":"8436bcc8be5f82f0e9b569b3d005bee71a163c1f41fba471f01ea19b953d9bc7"}
Nov 04 00:26:03.554140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.554106    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:26:03.631077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.630987    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca\") pod \"65935fe5-8739-4fe3-b88b-567944039c03\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") "
Nov 04 00:26:03.631321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.631130    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6\" (UniqueName: \"kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6\") pod \"65935fe5-8739-4fe3-b88b-567944039c03\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") "
Nov 04 00:26:03.631395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.631328    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca\") pod \"65935fe5-8739-4fe3-b88b-567944039c03\" (UID: \"65935fe5-8739-4fe3-b88b-567944039c03\") "
Nov 04 00:26:03.631457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.631410    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca" pod "65935fe5-8739-4fe3-b88b-567944039c03" (UID: "65935fe5-8739-4fe3-b88b-567944039c03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:03.631892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.631873    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:03.633658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.633465    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca" pod "65935fe5-8739-4fe3-b88b-567944039c03" (UID: "65935fe5-8739-4fe3-b88b-567944039c03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:03.634341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.633959    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6" pod "65935fe5-8739-4fe3-b88b-567944039c03" (UID: "65935fe5-8739-4fe3-b88b-567944039c03"). InnerVolumeSpecName "kube-api-access-6fgj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:03.734147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.734110    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65935fe5-8739-4fe3-b88b-567944039c03-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:03.734147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:03.734143    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgj6\" (UniqueName: \"kubernetes.io/projected/65935fe5-8739-4fe3-b88b-567944039c03-kube-api-access-6fgj6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:04.358009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:04.357968    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"
Nov 04 00:26:04.358009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:04.357996    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs" event={"ID":"65935fe5-8739-4fe3-b88b-567944039c03","Type":"ContainerDied","Data":"8d4fe8db73258f89fe80da04fb3ae4c62a262359ba7ec0c748aab1cd077d73f9"}
Nov 04 00:26:04.358270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:04.358081    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d4fe8db73258f89fe80da04fb3ae4c62a262359ba7ec0c748aab1cd077d73f9"
Nov 04 00:26:05.038913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:05.038875    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"]
Nov 04 00:26:05.050784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:05.048141    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"]
Nov 04 00:26:10.040038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.039991    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerName="payload-processing" containerID="cri-o://f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd" gracePeriod=25
Nov 04 00:26:10.048904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.048866    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" containerName="payload-pre-processing" containerID="cri-o://0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1" gracePeriod=25
Nov 04 00:26:10.843544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.843499    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"]
Nov 04 00:26:10.844511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.844487    5074 cpu_manager.go:414] "containerMap: removing container" podUID="65935fe5-8739-4fe3-b88b-567944039c03" containerName="revoke-keys"
Nov 04 00:26:10.844599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.844512    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65935fe5-8739-4fe3-b88b-567944039c03" containerName="revoke-keys"
Nov 04 00:26:10.844669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.844659    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="65935fe5-8739-4fe3-b88b-567944039c03" containerName="revoke-keys"
Nov 04 00:26:10.848952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.848920    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:10.856444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.856414    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:26:10.856698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.856681    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:26:10.858320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:10.858293    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"]
Nov 04 00:26:11.001471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.001431    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpbh\" (UniqueName: \"kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.001691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.001669    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.001783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.001712    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.103798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.103692    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.103798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.103779    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.104343 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.103960    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpbh\" (UniqueName: \"kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.104616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.104586    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.106330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.106306    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.117612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.117583    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpbh\" (UniqueName: \"kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh\") pod \"maas-api-revoke-sub-33c0a7cf-5sgwd\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.168258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.168224    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:11.509280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:11.509242    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"]
Nov 04 00:26:12.525474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:12.525446    5074 generic.go:356] "Generic (PLEG): container finished" podID="dcbe28bb-5abb-42f2-b330-5a05cf5e4375" containerID="93102fc8f1dcbe74380c6f6424d9bf5f947c5d619369a779d334c8b01bd33ad4" exitCode=0
Nov 04 00:26:12.525878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:12.525527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd" event={"ID":"dcbe28bb-5abb-42f2-b330-5a05cf5e4375","Type":"ContainerDied","Data":"93102fc8f1dcbe74380c6f6424d9bf5f947c5d619369a779d334c8b01bd33ad4"}
Nov 04 00:26:12.525878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:12.525628    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd" event={"ID":"dcbe28bb-5abb-42f2-b330-5a05cf5e4375","Type":"ContainerStarted","Data":"42775683f3a3eeec597aedfb1089b63f9c4f6c106ecdf9e46946b94c5207c4cf"}
Nov 04 00:26:13.767818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.767789    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:13.864174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.864080    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca\") pod \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") "
Nov 04 00:26:13.864356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.864202    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca\") pod \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") "
Nov 04 00:26:13.864356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.864328    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh\" (UniqueName: \"kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh\") pod \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\" (UID: \"dcbe28bb-5abb-42f2-b330-5a05cf5e4375\") "
Nov 04 00:26:13.864488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.864449    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca" pod "dcbe28bb-5abb-42f2-b330-5a05cf5e4375" (UID: "dcbe28bb-5abb-42f2-b330-5a05cf5e4375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:13.864897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.864877    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:13.866333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.866309    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca" pod "dcbe28bb-5abb-42f2-b330-5a05cf5e4375" (UID: "dcbe28bb-5abb-42f2-b330-5a05cf5e4375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:13.866500 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.866479    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh" pod "dcbe28bb-5abb-42f2-b330-5a05cf5e4375" (UID: "dcbe28bb-5abb-42f2-b330-5a05cf5e4375"). InnerVolumeSpecName "kube-api-access-cxpbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:13.966192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.966145    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpbh\" (UniqueName: \"kubernetes.io/projected/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-kube-api-access-cxpbh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:13.966192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:13.966192    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe28bb-5abb-42f2-b330-5a05cf5e4375-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:14.546646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:14.546613    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"
Nov 04 00:26:14.546875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:14.546617    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd" event={"ID":"dcbe28bb-5abb-42f2-b330-5a05cf5e4375","Type":"ContainerDied","Data":"42775683f3a3eeec597aedfb1089b63f9c4f6c106ecdf9e46946b94c5207c4cf"}
Nov 04 00:26:14.546875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:14.546752    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42775683f3a3eeec597aedfb1089b63f9c4f6c106ecdf9e46946b94c5207c4cf"
Nov 04 00:26:15.705333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:15.705297    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"]
Nov 04 00:26:15.706293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:15.706254    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy" containerID="cri-o://e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8" gracePeriod=30
Nov 04 00:26:15.715115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:15.715042    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"]
Nov 04 00:26:15.715691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:15.715639    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy" containerID="cri-o://3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e" gracePeriod=30
Nov 04 00:26:16.672434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:16.672385    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:26:16.673372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:16.673339    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:26:21.069550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.069513    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:26:21.091459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.091420    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:26:21.207861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.207824    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.207890    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.207976    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208030    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208094    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208147    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208184    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208204    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208234    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208266    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208331    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208384    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208432    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208480    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208526    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208584    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.208837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208618    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data\") pod \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\" (UID: \"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04\") "
Nov 04 00:26:21.208837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208657    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy\") pod \"c64c0883-ee43-46e9-baef-3a72361aced3\" (UID: \"c64c0883-ee43-46e9-baef-3a72361aced3\") "
Nov 04 00:26:21.209326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208933    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.208963    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.209154    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.209212    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.209366    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.209463    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.209535    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.209754 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.209687    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.210034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210008    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210249    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210275    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210297    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64c0883-ee43-46e9-baef-3a72361aced3-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210316    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210332    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210345    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210368    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210382    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.210457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210395    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.211191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.210515    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:21.212518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.212493    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "kube-api-access-gp8lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.212626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.212604    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.212756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.212717    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.212903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.212864    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:21.213197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.213166    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:21.213612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.213579    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6" pod "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" (UID: "dc9bec50-4989-4a9f-a4ce-46cc01ef5b04"). InnerVolumeSpecName "kube-api-access-gl5k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.214553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.214531    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.214637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.214572    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token" pod "c64c0883-ee43-46e9-baef-3a72361aced3" (UID: "c64c0883-ee43-46e9-baef-3a72361aced3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.311993    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312024    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312041    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64c0883-ee43-46e9-baef-3a72361aced3-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312055    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312069    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312084    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64c0883-ee43-46e9-baef-3a72361aced3-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312096    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312111    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp8lv\" (UniqueName: \"kubernetes.io/projected/c64c0883-ee43-46e9-baef-3a72361aced3-kube-api-access-gp8lv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.312507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.312123    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl5k6\" (UniqueName: \"kubernetes.io/projected/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04-kube-api-access-gl5k6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.653871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.650541    5074 generic.go:356] "Generic (PLEG): container finished" podID="c64c0883-ee43-46e9-baef-3a72361aced3" containerID="3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e" exitCode=0
Nov 04 00:26:21.653871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.650979    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" event={"ID":"c64c0883-ee43-46e9-baef-3a72361aced3","Type":"ContainerDied","Data":"3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e"}
Nov 04 00:26:21.653871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.651036    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc" event={"ID":"c64c0883-ee43-46e9-baef-3a72361aced3","Type":"ContainerDied","Data":"34b802ca8fcf333e8260f15f4db2e4e152ebf083bb6786cf12807597e012a4b3"}
Nov 04 00:26:21.653871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.651066    5074 scope.go:122] "RemoveContainer" containerID="3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e"
Nov 04 00:26:21.653871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.651244    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"
Nov 04 00:26:21.670074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.665629    5074 generic.go:356] "Generic (PLEG): container finished" podID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerID="e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8" exitCode=0
Nov 04 00:26:21.670074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.665909    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"
Nov 04 00:26:21.670074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.669935    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" event={"ID":"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04","Type":"ContainerDied","Data":"e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8"}
Nov 04 00:26:21.670074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.670008    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s" event={"ID":"dc9bec50-4989-4a9f-a4ce-46cc01ef5b04","Type":"ContainerDied","Data":"df7a4c78b8153a766da9f203829fa4b6385b8ebaca0be0c002b8f2db255d803b"}
Nov 04 00:26:21.678657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.678516    5074 scope.go:122] "RemoveContainer" containerID="3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e"
Nov 04 00:26:21.679297 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:21.679266    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e\": container with ID starting with 3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e not found: ID does not exist" containerID="3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e"
Nov 04 00:26:21.679425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.679304    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e"} err="failed to get container status \"3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e\": rpc error: code = NotFound desc = could not find container \"3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e\": container with ID starting with 3e2464b9ef2b5feb5688299a496ebca507f5a4da8229ecf9f8e124d5b902603e not found: ID does not exist"
Nov 04 00:26:21.679425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.679326    5074 scope.go:122] "RemoveContainer" containerID="e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8"
Nov 04 00:26:21.692971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.692940    5074 scope.go:122] "RemoveContainer" containerID="e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8"
Nov 04 00:26:21.693284 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:21.693260    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8\": container with ID starting with e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8 not found: ID does not exist" containerID="e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8"
Nov 04 00:26:21.693364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.693296    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8"} err="failed to get container status \"e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8\": rpc error: code = NotFound desc = could not find container \"e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8\": container with ID starting with e6d74b38bb11eab16efcaa0a004c429fd58a9bc371d5620d4345b6799f2b5aa8 not found: ID does not exist"
Nov 04 00:26:21.703516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.703483    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"]
Nov 04 00:26:21.711199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.711167    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-cjl5s"]
Nov 04 00:26:21.737819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.737782    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"]
Nov 04 00:26:21.767203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:21.767169    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-hcfcc"]
Nov 04 00:26:23.024281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:23.024241    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" path="/var/lib/kubelet/pods/c64c0883-ee43-46e9-baef-3a72361aced3/volumes"
Nov 04 00:26:23.025022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:23.025000    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" path="/var/lib/kubelet/pods/dc9bec50-4989-4a9f-a4ce-46cc01ef5b04/volumes"
Nov 04 00:26:26.670619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:26.670583    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:26:26.670619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:26.670605    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:26:28.024838 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:28.024784    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:26:29.184425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:29.184384    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"]
Nov 04 00:26:29.191567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:29.191536    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"]
Nov 04 00:26:30.354163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.354123    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:26:30.418153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.418117    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:26:30.465071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.465040    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume\") pod \"2c9a1d8f-0b21-4adb-9038-8b804224a320\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") "
Nov 04 00:26:30.465266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.465248    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt\" (UniqueName: \"kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt\") pod \"2c9a1d8f-0b21-4adb-9038-8b804224a320\" (UID: \"2c9a1d8f-0b21-4adb-9038-8b804224a320\") "
Nov 04 00:26:30.465462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.465440    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume" pod "2c9a1d8f-0b21-4adb-9038-8b804224a320" (UID: "2c9a1d8f-0b21-4adb-9038-8b804224a320"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:30.466065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.466035    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9a1d8f-0b21-4adb-9038-8b804224a320-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:30.467725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.467698    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt" pod "2c9a1d8f-0b21-4adb-9038-8b804224a320" (UID: "2c9a1d8f-0b21-4adb-9038-8b804224a320"). InnerVolumeSpecName "kube-api-access-9nxlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:30.567552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.567469    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume\") pod \"db489721-d8af-42cd-97df-49f2b24aa01c\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") "
Nov 04 00:26:30.567770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.567638    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6\" (UniqueName: \"kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6\") pod \"db489721-d8af-42cd-97df-49f2b24aa01c\" (UID: \"db489721-d8af-42cd-97df-49f2b24aa01c\") "
Nov 04 00:26:30.567899 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.567877    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume" pod "db489721-d8af-42cd-97df-49f2b24aa01c" (UID: "db489721-d8af-42cd-97df-49f2b24aa01c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:30.568224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.568199    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxlt\" (UniqueName: \"kubernetes.io/projected/2c9a1d8f-0b21-4adb-9038-8b804224a320-kube-api-access-9nxlt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:30.568268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.568236    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db489721-d8af-42cd-97df-49f2b24aa01c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:30.569804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.569783    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6" pod "db489721-d8af-42cd-97df-49f2b24aa01c" (UID: "db489721-d8af-42cd-97df-49f2b24aa01c"). InnerVolumeSpecName "kube-api-access-bmsc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:30.670226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.670183    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmsc6\" (UniqueName: \"kubernetes.io/projected/db489721-d8af-42cd-97df-49f2b24aa01c-kube-api-access-bmsc6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:30.768475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.768441    5074 generic.go:356] "Generic (PLEG): container finished" podID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerID="f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd" exitCode=0
Nov 04 00:26:30.768676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.768481    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"
Nov 04 00:26:30.768676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.768515    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" event={"ID":"2c9a1d8f-0b21-4adb-9038-8b804224a320","Type":"ContainerDied","Data":"f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd"}
Nov 04 00:26:30.768676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.768571    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p" event={"ID":"2c9a1d8f-0b21-4adb-9038-8b804224a320","Type":"ContainerDied","Data":"76abd0c63c36478565ca3b407d617bfec40452b73c1af253355bf2171b0d2bc3"}
Nov 04 00:26:30.768676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.768600    5074 scope.go:122] "RemoveContainer" containerID="f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd"
Nov 04 00:26:30.770310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.770285    5074 generic.go:356] "Generic (PLEG): container finished" podID="db489721-d8af-42cd-97df-49f2b24aa01c" containerID="0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1" exitCode=0
Nov 04 00:26:30.770459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.770324    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"
Nov 04 00:26:30.770459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.770325    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" event={"ID":"db489721-d8af-42cd-97df-49f2b24aa01c","Type":"ContainerDied","Data":"0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1"}
Nov 04 00:26:30.770459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.770431    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8" event={"ID":"db489721-d8af-42cd-97df-49f2b24aa01c","Type":"ContainerDied","Data":"1b9eb355af5a9c38f9e76c2fb07399d4ea029e7662ea2df16dfe8b4d67d8bce7"}
Nov 04 00:26:30.788702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.788682    5074 scope.go:122] "RemoveContainer" containerID="f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd"
Nov 04 00:26:30.789120 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:30.789095    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd\": container with ID starting with f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd not found: ID does not exist" containerID="f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd"
Nov 04 00:26:30.789208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.789128    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd"} err="failed to get container status \"f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd\": rpc error: code = NotFound desc = could not find container \"f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd\": container with ID starting with f27d2892081dd13a14ef33294d8a4f8aa85ebe049a3e8e2f407644ad76fa16bd not found: ID does not exist"
Nov 04 00:26:30.789208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.789147    5074 scope.go:122] "RemoveContainer" containerID="0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1"
Nov 04 00:26:30.800704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.800677    5074 scope.go:122] "RemoveContainer" containerID="0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1"
Nov 04 00:26:30.802024 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:30.801985    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1\": container with ID starting with 0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1 not found: ID does not exist" containerID="0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1"
Nov 04 00:26:30.802147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.802032    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1"} err="failed to get container status \"0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1\": rpc error: code = NotFound desc = could not find container \"0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1\": container with ID starting with 0ee560dd69cdadb81d3828f102ef8ce547edc8c7322825501a9240ab3276c4c1 not found: ID does not exist"
Nov 04 00:26:30.811966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.810750    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"]
Nov 04 00:26:30.813867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.813837    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85ad28-76fb45dd78-tkm2p"]
Nov 04 00:26:30.827923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.827841    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"]
Nov 04 00:26:30.832185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:30.832158    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85ad28-7f7cb6d7-gfsc8"]
Nov 04 00:26:31.019952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:31.019919    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" path="/var/lib/kubelet/pods/2c9a1d8f-0b21-4adb-9038-8b804224a320/volumes"
Nov 04 00:26:31.020402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:31.020388    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" path="/var/lib/kubelet/pods/db489721-d8af-42cd-97df-49f2b24aa01c/volumes"
Nov 04 00:26:34.185757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.185672    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx" podUID="59b9616a-cc75-4b14-9af5-84c6bfd16cb7" containerName="payload-processing" containerID="cri-o://685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26" gracePeriod=25
Nov 04 00:26:34.192232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.192199    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" podUID="f7688723-0aea-4153-959e-4473873f281e" containerName="payload-pre-processing" containerID="cri-o://72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366" gracePeriod=25
Nov 04 00:26:34.496593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.496568    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:26:34.526083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.526041    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk\" (UniqueName: \"kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk\") pod \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") "
Nov 04 00:26:34.526244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.526232    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume\") pod \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\" (UID: \"59b9616a-cc75-4b14-9af5-84c6bfd16cb7\") "
Nov 04 00:26:34.526669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.526643    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume" pod "59b9616a-cc75-4b14-9af5-84c6bfd16cb7" (UID: "59b9616a-cc75-4b14-9af5-84c6bfd16cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:34.526971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.526949    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:34.528617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.528588    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk" pod "59b9616a-cc75-4b14-9af5-84c6bfd16cb7" (UID: "59b9616a-cc75-4b14-9af5-84c6bfd16cb7"). InnerVolumeSpecName "kube-api-access-4q5qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:34.628469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.628435    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q5qk\" (UniqueName: \"kubernetes.io/projected/59b9616a-cc75-4b14-9af5-84c6bfd16cb7-kube-api-access-4q5qk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:34.832114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.832027    5074 generic.go:356] "Generic (PLEG): container finished" podID="59b9616a-cc75-4b14-9af5-84c6bfd16cb7" containerID="685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26" exitCode=0
Nov 04 00:26:34.832114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.832075    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"
Nov 04 00:26:34.832308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.832105    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx" event={"ID":"59b9616a-cc75-4b14-9af5-84c6bfd16cb7","Type":"ContainerDied","Data":"685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26"}
Nov 04 00:26:34.832308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.832175    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx" event={"ID":"59b9616a-cc75-4b14-9af5-84c6bfd16cb7","Type":"ContainerDied","Data":"993ba1028474f6fdc7ded96a90a2723bc65550d06a068e2b9af729f2794a06e5"}
Nov 04 00:26:34.832308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.832205    5074 scope.go:122] "RemoveContainer" containerID="685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26"
Nov 04 00:26:34.844635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.844613    5074 scope.go:122] "RemoveContainer" containerID="685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26"
Nov 04 00:26:34.844956 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:34.844930    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26\": container with ID starting with 685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26 not found: ID does not exist" containerID="685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26"
Nov 04 00:26:34.845046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.844965    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26"} err="failed to get container status \"685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26\": rpc error: code = NotFound desc = could not find container \"685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26\": container with ID starting with 685746c4ccfcc642df8be213a8d8a0a28560ac818e6874a705dc9de57945bf26 not found: ID does not exist"
Nov 04 00:26:34.861450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.861413    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"]
Nov 04 00:26:34.865715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:34.865694    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d94e94-d8978b8f8-s6mvx"]
Nov 04 00:26:35.022561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:35.022527    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b9616a-cc75-4b14-9af5-84c6bfd16cb7" path="/var/lib/kubelet/pods/59b9616a-cc75-4b14-9af5-84c6bfd16cb7/volumes"
Nov 04 00:26:35.827578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:35.827536    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" podUID="f7688723-0aea-4153-959e-4473873f281e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:26:45.827717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:45.827669    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" podUID="f7688723-0aea-4153-959e-4473873f281e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:26:53.398775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.398697    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"]
Nov 04 00:26:53.399633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399610    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy"
Nov 04 00:26:53.399633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399633    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399655    5074 cpu_manager.go:414] "containerMap: removing container" podUID="59b9616a-cc75-4b14-9af5-84c6bfd16cb7" containerName="payload-processing"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399663    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b9616a-cc75-4b14-9af5-84c6bfd16cb7" containerName="payload-processing"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399681    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399688    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399698    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe28bb-5abb-42f2-b330-5a05cf5e4375" containerName="revoke-keys"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399703    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe28bb-5abb-42f2-b330-5a05cf5e4375" containerName="revoke-keys"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399714    5074 cpu_manager.go:414] "containerMap: removing container" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" containerName="payload-pre-processing"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399719    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" containerName="payload-pre-processing"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399731    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerName="payload-processing"
Nov 04 00:26:53.399878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399750    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerName="payload-processing"
Nov 04 00:26:53.400397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399902    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b9616a-cc75-4b14-9af5-84c6bfd16cb7" containerName="payload-processing"
Nov 04 00:26:53.400397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399911    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9bec50-4989-4a9f-a4ce-46cc01ef5b04" containerName="istio-proxy"
Nov 04 00:26:53.400397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399918    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe28bb-5abb-42f2-b330-5a05cf5e4375" containerName="revoke-keys"
Nov 04 00:26:53.400397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399927    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="db489721-d8af-42cd-97df-49f2b24aa01c" containerName="payload-pre-processing"
Nov 04 00:26:53.400397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399934    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9a1d8f-0b21-4adb-9038-8b804224a320" containerName="payload-processing"
Nov 04 00:26:53.400397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.399947    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64c0883-ee43-46e9-baef-3a72361aced3" containerName="istio-proxy"
Nov 04 00:26:53.418821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.410011    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.418821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.413579    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"]
Nov 04 00:26:53.419703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.419625    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:26:53.440767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.433776    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.444959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.444919    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.468760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.465092    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwm8\" (UniqueName: \"kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.468760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.444763    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:26:53.589874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.576906    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.589874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.576984    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.589874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.577265    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwm8\" (UniqueName: \"kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.589874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.587108    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.589874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.587636    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.697827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.681871    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwm8\" (UniqueName: \"kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8\") pod \"maas-api-revoke-sub-e199e093-kd5qj\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:53.728761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.728521    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"]
Nov 04 00:26:53.728969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.728803    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy" containerID="cri-o://21ae62c9a7f6d97adedcab97a82b7efa21d742b1377ba9273f4f50d72f3ca862" gracePeriod=30
Nov 04 00:26:53.958091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:53.957982    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:54.438428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.437222    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"]
Nov 04 00:26:54.494094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.494068    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:26:54.543767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.539843    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f\" (UniqueName: \"kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f\") pod \"f7688723-0aea-4153-959e-4473873f281e\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") "
Nov 04 00:26:54.543767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.539905    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume\") pod \"f7688723-0aea-4153-959e-4473873f281e\" (UID: \"f7688723-0aea-4153-959e-4473873f281e\") "
Nov 04 00:26:54.543767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.542588    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume" pod "f7688723-0aea-4153-959e-4473873f281e" (UID: "f7688723-0aea-4153-959e-4473873f281e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:54.543767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.542939    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7688723-0aea-4153-959e-4473873f281e-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:54.544471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.544437    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f" pod "f7688723-0aea-4153-959e-4473873f281e" (UID: "f7688723-0aea-4153-959e-4473873f281e"). InnerVolumeSpecName "kube-api-access-2km7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:54.644439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:54.644397    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2km7f\" (UniqueName: \"kubernetes.io/projected/f7688723-0aea-4153-959e-4473873f281e-kube-api-access-2km7f\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:55.096972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.096925    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj" event={"ID":"d091ca77-9bad-4ace-860c-95fb01d4b06c","Type":"ContainerStarted","Data":"de46e6ae8347ebe05adab165e925c509391eb7bf6782d4ab85e9fefec82f6c06"}
Nov 04 00:26:55.097184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.097003    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj" event={"ID":"d091ca77-9bad-4ace-860c-95fb01d4b06c","Type":"ContainerStarted","Data":"bfe0bcd5f2e8619aefe91dfbbbb1b91abe83a9ef7b5a30f7cefaeb0f05ecf93e"}
Nov 04 00:26:55.099776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.098768    5074 generic.go:356] "Generic (PLEG): container finished" podID="f7688723-0aea-4153-959e-4473873f281e" containerID="72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366" exitCode=0
Nov 04 00:26:55.099776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.098820    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"
Nov 04 00:26:55.099776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.098862    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" event={"ID":"f7688723-0aea-4153-959e-4473873f281e","Type":"ContainerDied","Data":"72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366"}
Nov 04 00:26:55.099776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.098921    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs" event={"ID":"f7688723-0aea-4153-959e-4473873f281e","Type":"ContainerDied","Data":"a19ea464cc25419cbafe7993e0fd73176fce54de93204e46802393ad4aee74fd"}
Nov 04 00:26:55.099776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.098956    5074 scope.go:122] "RemoveContainer" containerID="72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366"
Nov 04 00:26:55.113695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.113665    5074 scope.go:122] "RemoveContainer" containerID="72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366"
Nov 04 00:26:55.114069 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:55.114041    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366\": container with ID starting with 72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366 not found: ID does not exist" containerID="72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366"
Nov 04 00:26:55.114181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.114078    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366"} err="failed to get container status \"72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366\": rpc error: code = NotFound desc = could not find container \"72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366\": container with ID starting with 72313c3b2a8ee77843144c10337b200138af2506fee3e1f3b6f0c08ca180f366 not found: ID does not exist"
Nov 04 00:26:55.122813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.122768    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj" podStartSLOduration=2.122751369 podStartE2EDuration="2.122751369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:26:55.118312783 +0000 UTC m=+3480.217912929" watchObservedRunningTime="2026-11-04 00:26:55.122751369 +0000 UTC m=+3480.222351513"
Nov 04 00:26:55.141204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.141172    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"]
Nov 04 00:26:55.147375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:55.147298    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d94e94-654859d75-k8jgs"]
Nov 04 00:26:56.124555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:56.124513    5074 generic.go:356] "Generic (PLEG): container finished" podID="d091ca77-9bad-4ace-860c-95fb01d4b06c" containerID="de46e6ae8347ebe05adab165e925c509391eb7bf6782d4ab85e9fefec82f6c06" exitCode=0
Nov 04 00:26:56.125128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:56.124564    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj" event={"ID":"d091ca77-9bad-4ace-860c-95fb01d4b06c","Type":"ContainerDied","Data":"de46e6ae8347ebe05adab165e925c509391eb7bf6782d4ab85e9fefec82f6c06"}
Nov 04 00:26:57.020209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.020173    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7688723-0aea-4153-959e-4473873f281e" path="/var/lib/kubelet/pods/f7688723-0aea-4153-959e-4473873f281e/volumes"
Nov 04 00:26:57.308944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.308912    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:57.381850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.381812    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8\" (UniqueName: \"kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8\") pod \"d091ca77-9bad-4ace-860c-95fb01d4b06c\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") "
Nov 04 00:26:57.382065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.381973    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca\") pod \"d091ca77-9bad-4ace-860c-95fb01d4b06c\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") "
Nov 04 00:26:57.382065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.382040    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca\") pod \"d091ca77-9bad-4ace-860c-95fb01d4b06c\" (UID: \"d091ca77-9bad-4ace-860c-95fb01d4b06c\") "
Nov 04 00:26:57.382354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.382326    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca" pod "d091ca77-9bad-4ace-860c-95fb01d4b06c" (UID: "d091ca77-9bad-4ace-860c-95fb01d4b06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:57.382625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.382605    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:57.384572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.384545    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca" pod "d091ca77-9bad-4ace-860c-95fb01d4b06c" (UID: "d091ca77-9bad-4ace-860c-95fb01d4b06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:57.384664 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.384586    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8" pod "d091ca77-9bad-4ace-860c-95fb01d4b06c" (UID: "d091ca77-9bad-4ace-860c-95fb01d4b06c"). InnerVolumeSpecName "kube-api-access-jdwm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:57.484372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.484328    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d091ca77-9bad-4ace-860c-95fb01d4b06c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:57.484372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:57.484369    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwm8\" (UniqueName: \"kubernetes.io/projected/d091ca77-9bad-4ace-860c-95fb01d4b06c-kube-api-access-jdwm8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:58.033299 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:26:58.033264    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:26:58.143571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:58.143540    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"
Nov 04 00:26:58.143782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:58.143569    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj" event={"ID":"d091ca77-9bad-4ace-860c-95fb01d4b06c","Type":"ContainerDied","Data":"bfe0bcd5f2e8619aefe91dfbbbb1b91abe83a9ef7b5a30f7cefaeb0f05ecf93e"}
Nov 04 00:26:58.143782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:58.143628    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe0bcd5f2e8619aefe91dfbbbb1b91abe83a9ef7b5a30f7cefaeb0f05ecf93e"
Nov 04 00:26:59.157273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.157138    5074 generic.go:356] "Generic (PLEG): container finished" podID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerID="21ae62c9a7f6d97adedcab97a82b7efa21d742b1377ba9273f4f50d72f3ca862" exitCode=0
Nov 04 00:26:59.157273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.157210    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" event={"ID":"430a67a5-71d7-4766-ad0f-bf046a861e53","Type":"ContainerDied","Data":"21ae62c9a7f6d97adedcab97a82b7efa21d742b1377ba9273f4f50d72f3ca862"}
Nov 04 00:26:59.355177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.355154    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:26:59.414406 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414373    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414474    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414510    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414640    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414719    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414781    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414831    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.414928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414837    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:59.415100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.414929    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.415100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415014    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket\") pod \"430a67a5-71d7-4766-ad0f-bf046a861e53\" (UID: \"430a67a5-71d7-4766-ad0f-bf046a861e53\") "
Nov 04 00:26:59.415100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415059    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:59.415369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415345    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:59.415990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415617    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.415990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415646    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.415990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415661    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.416260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.415989    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:59.416260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.416116    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:59.418048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.417620    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "kube-api-access-vzxh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:59.418174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.418110    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:59.418609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.418452    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:59.419193 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.419174    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy" pod "430a67a5-71d7-4766-ad0f-bf046a861e53" (UID: "430a67a5-71d7-4766-ad0f-bf046a861e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:59.516805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.516731    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.516805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.516808    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.516998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.516819    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.516998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.516829    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430a67a5-71d7-4766-ad0f-bf046a861e53-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.516998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.516839    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzxh7\" (UniqueName: \"kubernetes.io/projected/430a67a5-71d7-4766-ad0f-bf046a861e53-kube-api-access-vzxh7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:59.516998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:26:59.516849    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430a67a5-71d7-4766-ad0f-bf046a861e53-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:00.167974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:00.167941    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"
Nov 04 00:27:00.168432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:00.167981    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw" event={"ID":"430a67a5-71d7-4766-ad0f-bf046a861e53","Type":"ContainerDied","Data":"6786ffc77eae27bb2c73f59d14746b122777a43f2369719bdf7733af5c8966f9"}
Nov 04 00:27:00.168432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:00.168048    5074 scope.go:122] "RemoveContainer" containerID="21ae62c9a7f6d97adedcab97a82b7efa21d742b1377ba9273f4f50d72f3ca862"
Nov 04 00:27:00.195905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:00.195875    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"]
Nov 04 00:27:00.200555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:00.200525    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-n5prw"]
Nov 04 00:27:01.020699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:01.020613    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" path="/var/lib/kubelet/pods/430a67a5-71d7-4766-ad0f-bf046a861e53/volumes"
Nov 04 00:27:01.074675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:01.074631    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"]
Nov 04 00:27:01.084757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:01.084713    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"]
Nov 04 00:27:06.075278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.075220    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc" podUID="8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" containerName="payload-processing" containerID="cri-o://baecb5849a2ce461c6174dfb7e843870a2357c2414a9dd225fce4100471ce95c" gracePeriod=25
Nov 04 00:27:06.085058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.084997    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerName="payload-pre-processing" containerID="cri-o://c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c" gracePeriod=25
Nov 04 00:27:06.233507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.233467    5074 generic.go:356] "Generic (PLEG): container finished" podID="8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" containerID="baecb5849a2ce461c6174dfb7e843870a2357c2414a9dd225fce4100471ce95c" exitCode=0
Nov 04 00:27:06.233692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.233556    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc" event={"ID":"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c","Type":"ContainerDied","Data":"baecb5849a2ce461c6174dfb7e843870a2357c2414a9dd225fce4100471ce95c"}
Nov 04 00:27:06.400794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.400731    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:27:06.526946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.526902    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume\") pod \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") "
Nov 04 00:27:06.527166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.527147    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l\" (UniqueName: \"kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l\") pod \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\" (UID: \"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c\") "
Nov 04 00:27:06.527342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.527318    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume" pod "8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" (UID: "8b8e78b3-efc8-4b9e-8000-eefe6ca5889c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:06.527804 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.527781    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:06.529471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.529440    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l" pod "8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" (UID: "8b8e78b3-efc8-4b9e-8000-eefe6ca5889c"). InnerVolumeSpecName "kube-api-access-n7m4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:06.629420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.629318    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7m4l\" (UniqueName: \"kubernetes.io/projected/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c-kube-api-access-n7m4l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:06.653886 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:06.653846    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:27:07.246833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:07.246338    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"
Nov 04 00:27:07.246833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:07.246353    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc" event={"ID":"8b8e78b3-efc8-4b9e-8000-eefe6ca5889c","Type":"ContainerDied","Data":"fce4a38dee2b37a9d35e41b3c0e5b99c2c1401a18e76edda701c2eacbbf74c1c"}
Nov 04 00:27:07.246833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:07.246413    5074 scope.go:122] "RemoveContainer" containerID="baecb5849a2ce461c6174dfb7e843870a2357c2414a9dd225fce4100471ce95c"
Nov 04 00:27:07.269227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:07.269200    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"]
Nov 04 00:27:07.273434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:07.273406    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-757a7d-5c58cb9cdd-z7qzc"]
Nov 04 00:27:08.031268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:08.031232    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"]
Nov 04 00:27:08.037856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:08.037827    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca0a119-tfnc9"]
Nov 04 00:27:09.035435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:09.035395    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" path="/var/lib/kubelet/pods/8b8e78b3-efc8-4b9e-8000-eefe6ca5889c/volumes"
Nov 04 00:27:09.036153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:09.036130    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be991d81-1fd3-4c71-8895-ec5c34d163d9" path="/var/lib/kubelet/pods/be991d81-1fd3-4c71-8895-ec5c34d163d9/volumes"
Nov 04 00:27:16.653634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:16.653587    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:27:17.851184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.851143    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"]
Nov 04 00:27:17.852025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852005    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f7688723-0aea-4153-959e-4473873f281e" containerName="payload-pre-processing"
Nov 04 00:27:17.852080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852027    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7688723-0aea-4153-959e-4473873f281e" containerName="payload-pre-processing"
Nov 04 00:27:17.852080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852060    5074 cpu_manager.go:414] "containerMap: removing container" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy"
Nov 04 00:27:17.852080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852066    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy"
Nov 04 00:27:17.852196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852096    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d091ca77-9bad-4ace-860c-95fb01d4b06c" containerName="revoke-keys"
Nov 04 00:27:17.852196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852101    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d091ca77-9bad-4ace-860c-95fb01d4b06c" containerName="revoke-keys"
Nov 04 00:27:17.852196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852114    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" containerName="payload-processing"
Nov 04 00:27:17.852196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852122    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" containerName="payload-processing"
Nov 04 00:27:17.852321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852267    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7688723-0aea-4153-959e-4473873f281e" containerName="payload-pre-processing"
Nov 04 00:27:17.852321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852282    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="430a67a5-71d7-4766-ad0f-bf046a861e53" containerName="istio-proxy"
Nov 04 00:27:17.852321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852294    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d091ca77-9bad-4ace-860c-95fb01d4b06c" containerName="revoke-keys"
Nov 04 00:27:17.852321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.852301    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8e78b3-efc8-4b9e-8000-eefe6ca5889c" containerName="payload-processing"
Nov 04 00:27:17.858080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.858049    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:17.864888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.864832    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:27:17.879295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.865015    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:27:17.879295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.868782    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"]
Nov 04 00:27:17.948777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.948696    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:17.949028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.948933    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:17.949142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:17.949067    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2nj\" (UniqueName: \"kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.050483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.050440    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.050721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.050533    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2nj\" (UniqueName: \"kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.050721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.050637    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.051496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.051465    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.053184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.053160    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.064504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.064473    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2nj\" (UniqueName: \"kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj\") pod \"maas-api-revoke-sub-f20b8b04-mmlcj\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.183225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.183193    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:18.485125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:18.485092    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"]
Nov 04 00:27:18.485262 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:27:18.485178    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e5567f_f8b8_427d_8592_66c3c90a8c2c.slice/crio-8f031cd127576ff561467d491a07f90a08d22eff68739f1f08d8a007c257f7a9 WatchSource:0}: Error finding container 8f031cd127576ff561467d491a07f90a08d22eff68739f1f08d8a007c257f7a9: Status 404 returned error can't find the container with id 8f031cd127576ff561467d491a07f90a08d22eff68739f1f08d8a007c257f7a9
Nov 04 00:27:19.037062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:19.037030    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"]
Nov 04 00:27:19.041016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:19.040988    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4fc32-7jzxx"]
Nov 04 00:27:19.368603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:19.368553    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj" event={"ID":"71e5567f-f8b8-427d-8592-66c3c90a8c2c","Type":"ContainerStarted","Data":"cd2b389895b9d31d806193bc59324255b583e4d31067e1cc5f63c82e7876ad57"}
Nov 04 00:27:19.368821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:19.368638    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj" event={"ID":"71e5567f-f8b8-427d-8592-66c3c90a8c2c","Type":"ContainerStarted","Data":"8f031cd127576ff561467d491a07f90a08d22eff68739f1f08d8a007c257f7a9"}
Nov 04 00:27:19.389140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:19.389076    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj" podStartSLOduration=2.389060639 podStartE2EDuration="2.389060639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:19.387951624 +0000 UTC m=+3504.487551768" watchObservedRunningTime="2026-11-04 00:27:19.389060639 +0000 UTC m=+3504.488660783"
Nov 04 00:27:20.379337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:20.379300    5074 generic.go:356] "Generic (PLEG): container finished" podID="71e5567f-f8b8-427d-8592-66c3c90a8c2c" containerID="cd2b389895b9d31d806193bc59324255b583e4d31067e1cc5f63c82e7876ad57" exitCode=0
Nov 04 00:27:20.379760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:20.379347    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj" event={"ID":"71e5567f-f8b8-427d-8592-66c3c90a8c2c","Type":"ContainerDied","Data":"cd2b389895b9d31d806193bc59324255b583e4d31067e1cc5f63c82e7876ad57"}
Nov 04 00:27:21.023292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.023248    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7e4d27-0ada-44d6-989a-098a33b5a4f4" path="/var/lib/kubelet/pods/ba7e4d27-0ada-44d6-989a-098a33b5a4f4/volumes"
Nov 04 00:27:21.569153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.569130    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:21.618598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.609860    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj\" (UniqueName: \"kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj\") pod \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") "
Nov 04 00:27:21.618598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.609943    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca\") pod \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") "
Nov 04 00:27:21.618598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.609979    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca\") pod \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\" (UID: \"71e5567f-f8b8-427d-8592-66c3c90a8c2c\") "
Nov 04 00:27:21.618598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.611228    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca" pod "71e5567f-f8b8-427d-8592-66c3c90a8c2c" (UID: "71e5567f-f8b8-427d-8592-66c3c90a8c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:21.618598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.613879    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca" pod "71e5567f-f8b8-427d-8592-66c3c90a8c2c" (UID: "71e5567f-f8b8-427d-8592-66c3c90a8c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:21.618598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.614017    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj" pod "71e5567f-f8b8-427d-8592-66c3c90a8c2c" (UID: "71e5567f-f8b8-427d-8592-66c3c90a8c2c"). InnerVolumeSpecName "kube-api-access-pb2nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:21.712509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.712475    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2nj\" (UniqueName: \"kubernetes.io/projected/71e5567f-f8b8-427d-8592-66c3c90a8c2c-kube-api-access-pb2nj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:21.712509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.712509    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:21.712711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:21.712523    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e5567f-f8b8-427d-8592-66c3c90a8c2c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:22.398177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:22.398146    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"
Nov 04 00:27:22.398359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:22.398172    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj" event={"ID":"71e5567f-f8b8-427d-8592-66c3c90a8c2c","Type":"ContainerDied","Data":"8f031cd127576ff561467d491a07f90a08d22eff68739f1f08d8a007c257f7a9"}
Nov 04 00:27:22.398359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:22.398234    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f031cd127576ff561467d491a07f90a08d22eff68739f1f08d8a007c257f7a9"
Nov 04 00:27:25.494684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.494641    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"]
Nov 04 00:27:25.496506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.496470    5074 cpu_manager.go:414] "containerMap: removing container" podUID="71e5567f-f8b8-427d-8592-66c3c90a8c2c" containerName="revoke-keys"
Nov 04 00:27:25.496506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.496502    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e5567f-f8b8-427d-8592-66c3c90a8c2c" containerName="revoke-keys"
Nov 04 00:27:25.496752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.496725    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e5567f-f8b8-427d-8592-66c3c90a8c2c" containerName="revoke-keys"
Nov 04 00:27:25.519173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.519139    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.539914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.539789    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c9c0227c-dockercfg-7pphz\""
Nov 04 00:27:25.546992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.546598    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"]
Nov 04 00:27:25.563625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.563587    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpb6\" (UniqueName: \"kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6\") pod \"payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") " pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.564033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.564003    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") " pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.629113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.629078    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c9c0227c\""
Nov 04 00:27:25.691989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.685949    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpb6\" (UniqueName: \"kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6\") pod \"payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") " pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.708774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.698845    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") " pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.708774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.700106    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") " pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.717367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.717334    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpb6\" (UniqueName: \"kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6\") pod \"payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") " pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:25.851291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:25.851209    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:26.152753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.152713    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"]
Nov 04 00:27:26.157138 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:27:26.157106    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71801ec_cc14_4c34_8d9b_6c13f5a2e0d3.slice/crio-030c0cea0d7fdd64b158df9ef603a014f93cd434e9d284c7b1951319d08e5e87 WatchSource:0}: Error finding container 030c0cea0d7fdd64b158df9ef603a014f93cd434e9d284c7b1951319d08e5e87: Status 404 returned error can't find the container with id 030c0cea0d7fdd64b158df9ef603a014f93cd434e9d284c7b1951319d08e5e87
Nov 04 00:27:26.367761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.367705    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:27:26.455071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.455035    5074 generic.go:356] "Generic (PLEG): container finished" podID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerID="c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c" exitCode=0
Nov 04 00:27:26.455259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.455142    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" event={"ID":"25fc0539-0811-46b1-92f2-7d1062b6f0d5","Type":"ContainerDied","Data":"c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c"}
Nov 04 00:27:26.455259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.455180    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv" event={"ID":"25fc0539-0811-46b1-92f2-7d1062b6f0d5","Type":"ContainerDied","Data":"1ab2bb0732d83efca92dc897feefb5330846e5ee54f17f1b029b914e3bef0859"}
Nov 04 00:27:26.455259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.455202    5074 scope.go:122] "RemoveContainer" containerID="c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c"
Nov 04 00:27:26.455390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.455321    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"
Nov 04 00:27:26.458511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.458476    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj" event={"ID":"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3","Type":"ContainerStarted","Data":"46d72d277097818c4904681e7d34390eff614ac5415688a04e55fafe34c60941"}
Nov 04 00:27:26.458641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.458540    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj" event={"ID":"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3","Type":"ContainerStarted","Data":"030c0cea0d7fdd64b158df9ef603a014f93cd434e9d284c7b1951319d08e5e87"}
Nov 04 00:27:26.458641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.458590    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:26.469070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.469047    5074 scope.go:122] "RemoveContainer" containerID="c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c"
Nov 04 00:27:26.469394 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:27:26.469371    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c\": container with ID starting with c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c not found: ID does not exist" containerID="c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c"
Nov 04 00:27:26.469470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.469401    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c"} err="failed to get container status \"c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c\": rpc error: code = NotFound desc = could not find container \"c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c\": container with ID starting with c4c90718f7c18facc9b65752ba18983149dbcaa9b1c559b95b55e654085d9d6c not found: ID does not exist"
Nov 04 00:27:26.481786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.481688    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj" podStartSLOduration=1.481669985 podStartE2EDuration="1.481669985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:26.478597521 +0000 UTC m=+3511.578197666" watchObservedRunningTime="2026-11-04 00:27:26.481669985 +0000 UTC m=+3511.581270131"
Nov 04 00:27:26.510509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.510465    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume\") pod \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") "
Nov 04 00:27:26.511094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.510538    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b\" (UniqueName: \"kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b\") pod \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\" (UID: \"25fc0539-0811-46b1-92f2-7d1062b6f0d5\") "
Nov 04 00:27:26.511094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.510909    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume" pod "25fc0539-0811-46b1-92f2-7d1062b6f0d5" (UID: "25fc0539-0811-46b1-92f2-7d1062b6f0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:26.511522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.511491    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25fc0539-0811-46b1-92f2-7d1062b6f0d5-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:26.513020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.512987    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b" pod "25fc0539-0811-46b1-92f2-7d1062b6f0d5" (UID: "25fc0539-0811-46b1-92f2-7d1062b6f0d5"). InnerVolumeSpecName "kube-api-access-s5w6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:26.612826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.612675    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5w6b\" (UniqueName: \"kubernetes.io/projected/25fc0539-0811-46b1-92f2-7d1062b6f0d5-kube-api-access-s5w6b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:26.785151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.785118    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"]
Nov 04 00:27:26.789285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:26.789258    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-757a7d-68654d4ff-42xdv"]
Nov 04 00:27:27.021909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:27.021873    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" path="/var/lib/kubelet/pods/25fc0539-0811-46b1-92f2-7d1062b6f0d5/volumes"
Nov 04 00:27:28.023592 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:27:28.023556    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:27:28.934648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.934583    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"]
Nov 04 00:27:28.935480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.935462    5074 cpu_manager.go:414] "containerMap: removing container" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerName="payload-pre-processing"
Nov 04 00:27:28.935480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.935479    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerName="payload-pre-processing"
Nov 04 00:27:28.935643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.935605    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="25fc0539-0811-46b1-92f2-7d1062b6f0d5" containerName="payload-pre-processing"
Nov 04 00:27:28.939824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.939750    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:28.947206 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.944724    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:27:28.947206 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.944878    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:27:28.948541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:28.948504    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"]
Nov 04 00:27:29.050653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.050536    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.072932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.050707    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.072932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.050842    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zm28\" (UniqueName: \"kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.152055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.152016    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm28\" (UniqueName: \"kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.152241 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.152152    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.152323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.152303    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.152922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.152892    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.154733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.154713    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.164713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.164686    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zm28\" (UniqueName: \"kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28\") pod \"maas-api-revoke-sub-9ffd8cfa-x6wsl\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.268031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.267939    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:29.590704 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:29.590670    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"]
Nov 04 00:27:29.590832 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:27:29.590769    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc6fe8f_9a21_424a_95b0_04342455a5f8.slice/crio-9f5c12f76980f1eb6e9be75f6302641e02650087988de6eadb2ff85b80e0f3ef WatchSource:0}: Error finding container 9f5c12f76980f1eb6e9be75f6302641e02650087988de6eadb2ff85b80e0f3ef: Status 404 returned error can't find the container with id 9f5c12f76980f1eb6e9be75f6302641e02650087988de6eadb2ff85b80e0f3ef
Nov 04 00:27:30.503218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:30.503186    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl" event={"ID":"fcc6fe8f-9a21-424a-95b0-04342455a5f8","Type":"ContainerStarted","Data":"72d463c6d2aa85f030408a3dc15a8989ba50bebb4352f33abbd49ce3e97d91e9"}
Nov 04 00:27:30.503800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:30.503244    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl" event={"ID":"fcc6fe8f-9a21-424a-95b0-04342455a5f8","Type":"ContainerStarted","Data":"9f5c12f76980f1eb6e9be75f6302641e02650087988de6eadb2ff85b80e0f3ef"}
Nov 04 00:27:30.526168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:30.526105    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl" podStartSLOduration=2.526090339 podStartE2EDuration="2.526090339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:30.522459778 +0000 UTC m=+3515.622059924" watchObservedRunningTime="2026-11-04 00:27:30.526090339 +0000 UTC m=+3515.625690529"
Nov 04 00:27:31.515714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:31.515672    5074 generic.go:356] "Generic (PLEG): container finished" podID="fcc6fe8f-9a21-424a-95b0-04342455a5f8" containerID="72d463c6d2aa85f030408a3dc15a8989ba50bebb4352f33abbd49ce3e97d91e9" exitCode=0
Nov 04 00:27:31.516198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:31.515715    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl" event={"ID":"fcc6fe8f-9a21-424a-95b0-04342455a5f8","Type":"ContainerDied","Data":"72d463c6d2aa85f030408a3dc15a8989ba50bebb4352f33abbd49ce3e97d91e9"}
Nov 04 00:27:32.733966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.733933    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:32.816168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.815029    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca\") pod \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") "
Nov 04 00:27:32.816168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.815235    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28\" (UniqueName: \"kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28\") pod \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") "
Nov 04 00:27:32.816168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.815312    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca\") pod \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\" (UID: \"fcc6fe8f-9a21-424a-95b0-04342455a5f8\") "
Nov 04 00:27:32.816820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.816550    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca" pod "fcc6fe8f-9a21-424a-95b0-04342455a5f8" (UID: "fcc6fe8f-9a21-424a-95b0-04342455a5f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:32.826263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.826222    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28" pod "fcc6fe8f-9a21-424a-95b0-04342455a5f8" (UID: "fcc6fe8f-9a21-424a-95b0-04342455a5f8"). InnerVolumeSpecName "kube-api-access-7zm28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:32.828602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.828570    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca" pod "fcc6fe8f-9a21-424a-95b0-04342455a5f8" (UID: "fcc6fe8f-9a21-424a-95b0-04342455a5f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:32.917565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.917525    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:32.917565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.917562    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc6fe8f-9a21-424a-95b0-04342455a5f8-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:32.917565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:32.917572    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zm28\" (UniqueName: \"kubernetes.io/projected/fcc6fe8f-9a21-424a-95b0-04342455a5f8-kube-api-access-7zm28\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:33.546063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:33.546025    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl" event={"ID":"fcc6fe8f-9a21-424a-95b0-04342455a5f8","Type":"ContainerDied","Data":"9f5c12f76980f1eb6e9be75f6302641e02650087988de6eadb2ff85b80e0f3ef"}
Nov 04 00:27:33.546063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:33.546055    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"
Nov 04 00:27:33.546291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:33.546089    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f5c12f76980f1eb6e9be75f6302641e02650087988de6eadb2ff85b80e0f3ef"
Nov 04 00:27:37.471518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:37.471488    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:27:42.799866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:42.799827    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"]
Nov 04 00:27:42.800281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:42.800040    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" podUID="f73e0d3d-6097-429e-b198-35447cfa028e" containerName="discovery" containerID="cri-o://acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f" gracePeriod=30
Nov 04 00:27:43.110147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.110118    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:27:43.195211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195175    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.195211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195218    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.195482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195267    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.195482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195457    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.195602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195570    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.195708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195687    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.195820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.195805    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs\") pod \"f73e0d3d-6097-429e-b198-35447cfa028e\" (UID: \"f73e0d3d-6097-429e-b198-35447cfa028e\") "
Nov 04 00:27:43.197713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.197667    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:43.201584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.201549    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:43.201852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.201812    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:43.201852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.201828    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:43.202009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.201851    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:43.202009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.201867    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:43.203482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.203450    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh" pod "f73e0d3d-6097-429e-b198-35447cfa028e" (UID: "f73e0d3d-6097-429e-b198-35447cfa028e"). InnerVolumeSpecName "kube-api-access-6l4wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:43.298534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298501    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.298534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298529    5074 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-cacerts\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.298534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298540    5074 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-kubeconfig\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.298793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298549    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l4wh\" (UniqueName: \"kubernetes.io/projected/f73e0d3d-6097-429e-b198-35447cfa028e-kube-api-access-6l4wh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.298793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298560    5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-dns-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.298793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298569    5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73e0d3d-6097-429e-b198-35447cfa028e-istio-csr-ca-configmap\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.298793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.298577    5074 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e0d3d-6097-429e-b198-35447cfa028e-local-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:43.644379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.644339    5074 generic.go:356] "Generic (PLEG): container finished" podID="f73e0d3d-6097-429e-b198-35447cfa028e" containerID="acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f" exitCode=0
Nov 04 00:27:43.644568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.644400    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"
Nov 04 00:27:43.644568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.644418    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" event={"ID":"f73e0d3d-6097-429e-b198-35447cfa028e","Type":"ContainerDied","Data":"acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f"}
Nov 04 00:27:43.644568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.644472    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4" event={"ID":"f73e0d3d-6097-429e-b198-35447cfa028e","Type":"ContainerDied","Data":"3755431fef49de730e0bed1e6eb544d86c2d03478960ff0effa949d49b482b32"}
Nov 04 00:27:43.644568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.644495    5074 scope.go:122] "RemoveContainer" containerID="acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f"
Nov 04 00:27:43.659364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.659343    5074 scope.go:122] "RemoveContainer" containerID="acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f"
Nov 04 00:27:43.659656 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:27:43.659630    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f\": container with ID starting with acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f not found: ID does not exist" containerID="acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f"
Nov 04 00:27:43.659755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.659663    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f"} err="failed to get container status \"acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f\": rpc error: code = NotFound desc = could not find container \"acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f\": container with ID starting with acc987a0e8263eaf60c69fc622a44b5396f2c00aab880b9723e009ea4bee627f not found: ID does not exist"
Nov 04 00:27:43.680137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.680109    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"]
Nov 04 00:27:43.686654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:43.686626    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pfdb4"]
Nov 04 00:27:45.023566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.023524    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73e0d3d-6097-429e-b198-35447cfa028e" path="/var/lib/kubelet/pods/f73e0d3d-6097-429e-b198-35447cfa028e/volumes"
Nov 04 00:27:45.219760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.219489    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"]
Nov 04 00:27:45.228764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.220658    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f73e0d3d-6097-429e-b198-35447cfa028e" containerName="discovery"
Nov 04 00:27:45.228764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.220684    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73e0d3d-6097-429e-b198-35447cfa028e" containerName="discovery"
Nov 04 00:27:45.228764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.220700    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fcc6fe8f-9a21-424a-95b0-04342455a5f8" containerName="revoke-keys"
Nov 04 00:27:45.228764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.220708    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc6fe8f-9a21-424a-95b0-04342455a5f8" containerName="revoke-keys"
Nov 04 00:27:45.228764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.220918    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73e0d3d-6097-429e-b198-35447cfa028e" containerName="discovery"
Nov 04 00:27:45.228764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.220931    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc6fe8f-9a21-424a-95b0-04342455a5f8" containerName="revoke-keys"
Nov 04 00:27:45.241789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.237022    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.243957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.243923    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:27:45.247924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.247901    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:27:45.269133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.259659    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"]
Nov 04 00:27:45.333604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.333507    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.333974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.333948    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4jt\" (UniqueName: \"kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.334262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.334229    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.435409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.435362    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4jt\" (UniqueName: \"kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.435639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.435517    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.435771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.435719    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.436140 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.436120    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.438103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.438077    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.448952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.448925    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4jt\" (UniqueName: \"kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt\") pod \"maas-api-revoke-sub-21ad313b-l8bmw\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.643908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.643807    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:45.794813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.794773    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"]
Nov 04 00:27:45.841897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.841860    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"]
Nov 04 00:27:45.960948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:45.960914    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"]
Nov 04 00:27:46.678414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:46.678324    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw" event={"ID":"36fce481-441d-4b3f-93e3-67646d320b00","Type":"ContainerStarted","Data":"ffe4585c828d00de75e79ede16eec857d83786827ba2142845f52ddb935dd568"}
Nov 04 00:27:46.678414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:46.678378    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw" event={"ID":"36fce481-441d-4b3f-93e3-67646d320b00","Type":"ContainerStarted","Data":"262bfd0be0af3e184f19adb85c7be4cbfed6e59d0d21df443b8dd60b6dc4c8b5"}
Nov 04 00:27:46.699665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:46.699600    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw" podStartSLOduration=1.699586494 podStartE2EDuration="1.699586494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:46.697031092 +0000 UTC m=+3531.796631249" watchObservedRunningTime="2026-11-04 00:27:46.699586494 +0000 UTC m=+3531.799186638"
Nov 04 00:27:47.268034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.267997    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"]
Nov 04 00:27:47.273077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.273051    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.284636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.284606    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"]
Nov 04 00:27:47.291948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.291115    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c9c0227c-serving-cert\""
Nov 04 00:27:47.299049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.298301    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.299049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.298366    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.299049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.298506    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdrk\" (UniqueName: \"kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.299049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.298538    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.406279    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.406354    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.406484    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdrk\" (UniqueName: \"kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.406514    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.407393    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:27:47.407504    5074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-serving-cert: secret "maas-api-e2e-mt-c9c0227c-serving-cert" not found
Nov 04 00:27:47.408652 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:27:47.407560    5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls podName:0879e1e4-f187-4442-bc2f-35b90eb8931a nodeName:}" failed. No retries permitted until 2026-11-04 00:27:47.907540945 +0000 UTC m=+3533.007141072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls") pod "maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" (UID: "0879e1e4-f187-4442-bc2f-35b90eb8931a") : secret "maas-api-e2e-mt-c9c0227c-serving-cert" not found
Nov 04 00:27:47.428416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.428382    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.432259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.432229    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdrk\" (UniqueName: \"kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.693365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.693329    5074 generic.go:356] "Generic (PLEG): container finished" podID="36fce481-441d-4b3f-93e3-67646d320b00" containerID="ffe4585c828d00de75e79ede16eec857d83786827ba2142845f52ddb935dd568" exitCode=0
Nov 04 00:27:47.693912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.693382    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw" event={"ID":"36fce481-441d-4b3f-93e3-67646d320b00","Type":"ContainerDied","Data":"ffe4585c828d00de75e79ede16eec857d83786827ba2142845f52ddb935dd568"}
Nov 04 00:27:47.917763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.916803    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:47.945968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:47.944932    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\") pod \"maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:48.212494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.212404    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:48.531223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.531118    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"]
Nov 04 00:27:48.531731 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:27:48.531699    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0879e1e4_f187_4442_bc2f_35b90eb8931a.slice/crio-1bdee0b9790045485a0349ddd08cddffb4d5264547e837832302f292d233402d WatchSource:0}: Error finding container 1bdee0b9790045485a0349ddd08cddffb4d5264547e837832302f292d233402d: Status 404 returned error can't find the container with id 1bdee0b9790045485a0349ddd08cddffb4d5264547e837832302f292d233402d
Nov 04 00:27:48.533510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.533483    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:27:48.704814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.704778    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" event={"ID":"0879e1e4-f187-4442-bc2f-35b90eb8931a","Type":"ContainerStarted","Data":"1bdee0b9790045485a0349ddd08cddffb4d5264547e837832302f292d233402d"}
Nov 04 00:27:48.878648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.878623    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:48.963835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.963793    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt\" (UniqueName: \"kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt\") pod \"36fce481-441d-4b3f-93e3-67646d320b00\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") "
Nov 04 00:27:48.964032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.963901    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca\") pod \"36fce481-441d-4b3f-93e3-67646d320b00\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") "
Nov 04 00:27:48.964137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.964122    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca\") pod \"36fce481-441d-4b3f-93e3-67646d320b00\" (UID: \"36fce481-441d-4b3f-93e3-67646d320b00\") "
Nov 04 00:27:48.964607 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.964574    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca" pod "36fce481-441d-4b3f-93e3-67646d320b00" (UID: "36fce481-441d-4b3f-93e3-67646d320b00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:48.966302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.966276    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt" pod "36fce481-441d-4b3f-93e3-67646d320b00" (UID: "36fce481-441d-4b3f-93e3-67646d320b00"). InnerVolumeSpecName "kube-api-access-qg4jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:48.966412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:48.966329    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca" pod "36fce481-441d-4b3f-93e3-67646d320b00" (UID: "36fce481-441d-4b3f-93e3-67646d320b00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:49.066935 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.066908    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:49.067042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.066941    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4jt\" (UniqueName: \"kubernetes.io/projected/36fce481-441d-4b3f-93e3-67646d320b00-kube-api-access-qg4jt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:49.067042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.066957    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fce481-441d-4b3f-93e3-67646d320b00-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:49.718282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.718232    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" event={"ID":"0879e1e4-f187-4442-bc2f-35b90eb8931a","Type":"ContainerStarted","Data":"726b8dd9474e96762dc7e5db8709419f6fa007362dfeea3711c71d33f657d186"}
Nov 04 00:27:49.718999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.718340    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:49.720793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.720290    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"
Nov 04 00:27:49.720793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.720361    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw" event={"ID":"36fce481-441d-4b3f-93e3-67646d320b00","Type":"ContainerDied","Data":"262bfd0be0af3e184f19adb85c7be4cbfed6e59d0d21df443b8dd60b6dc4c8b5"}
Nov 04 00:27:49.720793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.720501    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262bfd0be0af3e184f19adb85c7be4cbfed6e59d0d21df443b8dd60b6dc4c8b5"
Nov 04 00:27:49.750989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:49.750938    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" podStartSLOduration=2.22519411 podStartE2EDuration="2.750924564s" totalImagesPullingTime="525.730454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:49.741824454 +0000 UTC m=+3534.841424610" watchObservedRunningTime="2026-11-04 00:27:49.750924564 +0000 UTC m=+3534.850524707"
Nov 04 00:27:50.795417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:50.795367    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerName="payload-processing" containerID="cri-o://c026b764909785e24672d906fce87071c884e8ee2aeab52b32c5faeaef166845" gracePeriod=25
Nov 04 00:27:50.843493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:50.843447    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerName="payload-pre-processing" containerID="cri-o://006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979" gracePeriod=25
Nov 04 00:27:52.161422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:52.161381    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:27:52.161422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:52.161408    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:27:55.739766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:27:55.738749    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:27:58.092886 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:27:58.092844    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:28:02.161632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:02.161587    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:28:02.162122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:02.161594    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:28:10.954467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:10.954405    5074 generic.go:356] "Generic (PLEG): container finished" podID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerID="c026b764909785e24672d906fce87071c884e8ee2aeab52b32c5faeaef166845" exitCode=0
Nov 04 00:28:10.954919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:10.954532    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" event={"ID":"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d","Type":"ContainerDied","Data":"c026b764909785e24672d906fce87071c884e8ee2aeab52b32c5faeaef166845"}
Nov 04 00:28:11.155774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.155723    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:28:11.222168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.222137    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:28:11.258919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.258884    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76\" (UniqueName: \"kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76\") pod \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") "
Nov 04 00:28:11.259155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.259136    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume\") pod \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\" (UID: \"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d\") "
Nov 04 00:28:11.259615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.259586    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume" pod "0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" (UID: "0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:28:11.259872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.259851    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:11.261254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.261226    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76" pod "0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" (UID: "0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d"). InnerVolumeSpecName "kube-api-access-2qh76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:28:11.360941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.360862    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg\" (UniqueName: \"kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg\") pod \"12115daa-a907-406d-ba0d-1b8b664d24ef\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") "
Nov 04 00:28:11.361119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.361049    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume\") pod \"12115daa-a907-406d-ba0d-1b8b664d24ef\" (UID: \"12115daa-a907-406d-ba0d-1b8b664d24ef\") "
Nov 04 00:28:11.362120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.362071    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume" pod "12115daa-a907-406d-ba0d-1b8b664d24ef" (UID: "12115daa-a907-406d-ba0d-1b8b664d24ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:28:11.370782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.363917    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qh76\" (UniqueName: \"kubernetes.io/projected/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d-kube-api-access-2qh76\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:11.370782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.369301    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg" pod "12115daa-a907-406d-ba0d-1b8b664d24ef" (UID: "12115daa-a907-406d-ba0d-1b8b664d24ef"). InnerVolumeSpecName "kube-api-access-jzljg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:28:11.465461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.465429    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzljg\" (UniqueName: \"kubernetes.io/projected/12115daa-a907-406d-ba0d-1b8b664d24ef-kube-api-access-jzljg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:11.465575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.465467    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12115daa-a907-406d-ba0d-1b8b664d24ef-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:11.969457 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.969406    5074 generic.go:356] "Generic (PLEG): container finished" podID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerID="006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979" exitCode=0
Nov 04 00:28:11.970103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.969573    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" event={"ID":"12115daa-a907-406d-ba0d-1b8b664d24ef","Type":"ContainerDied","Data":"006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979"}
Nov 04 00:28:11.970103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.969665    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq" event={"ID":"12115daa-a907-406d-ba0d-1b8b664d24ef","Type":"ContainerDied","Data":"6b25fedf7ca759cbc111abdc3246317680588f5e126f637229a0f92279640963"}
Nov 04 00:28:11.970103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.969695    5074 scope.go:122] "RemoveContainer" containerID="006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979"
Nov 04 00:28:11.970103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.969888    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"
Nov 04 00:28:11.986793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.984086    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"
Nov 04 00:28:11.986793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.984111    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz" event={"ID":"0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d","Type":"ContainerDied","Data":"7c2aca4307487923a8e3cd2992e30d5c1eb06654aa2079e9aaa88cc1fcc5de99"}
Nov 04 00:28:11.996828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.996805    5074 scope.go:122] "RemoveContainer" containerID="006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979"
Nov 04 00:28:11.997161 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:11.997140    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979\": container with ID starting with 006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979 not found: ID does not exist" containerID="006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979"
Nov 04 00:28:11.997243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.997173    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979"} err="failed to get container status \"006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979\": rpc error: code = NotFound desc = could not find container \"006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979\": container with ID starting with 006f63290b08d49ac8cad07ca9e0e76a031412111efb2629b66b28edac1a9979 not found: ID does not exist"
Nov 04 00:28:11.997243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:11.997197    5074 scope.go:122] "RemoveContainer" containerID="c026b764909785e24672d906fce87071c884e8ee2aeab52b32c5faeaef166845"
Nov 04 00:28:12.015195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:12.015167    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"]
Nov 04 00:28:12.019589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:12.019562    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecd46b-b68bb9f8c-s8kjq"]
Nov 04 00:28:12.032016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:12.031984    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"]
Nov 04 00:28:12.036505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:12.036479    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd46b-86654cc996-744bz"]
Nov 04 00:28:13.022597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:13.022556    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" path="/var/lib/kubelet/pods/0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d/volumes"
Nov 04 00:28:13.023147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:13.023070    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" path="/var/lib/kubelet/pods/12115daa-a907-406d-ba0d-1b8b664d24ef/volumes"
Nov 04 00:28:20.154349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:20.154305    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"]
Nov 04 00:28:25.155356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.155309    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5" podUID="3da8a195-2855-42e0-8035-e8991fd5b0fd" containerName="payload-processing" containerID="cri-o://f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df" gracePeriod=25
Nov 04 00:28:25.455628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.455601    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:28:25.502884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.502846    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume\") pod \"3da8a195-2855-42e0-8035-e8991fd5b0fd\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") "
Nov 04 00:28:25.503143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.503124    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws\" (UniqueName: \"kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws\") pod \"3da8a195-2855-42e0-8035-e8991fd5b0fd\" (UID: \"3da8a195-2855-42e0-8035-e8991fd5b0fd\") "
Nov 04 00:28:25.520109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.520077    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws" pod "3da8a195-2855-42e0-8035-e8991fd5b0fd" (UID: "3da8a195-2855-42e0-8035-e8991fd5b0fd"). InnerVolumeSpecName "kube-api-access-4snws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:28:25.520733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.520706    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume" pod "3da8a195-2855-42e0-8035-e8991fd5b0fd" (UID: "3da8a195-2855-42e0-8035-e8991fd5b0fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:28:25.604482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.604445    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snws\" (UniqueName: \"kubernetes.io/projected/3da8a195-2855-42e0-8035-e8991fd5b0fd-kube-api-access-4snws\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:25.604482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:25.604477    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da8a195-2855-42e0-8035-e8991fd5b0fd-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:26.112755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.112702    5074 generic.go:356] "Generic (PLEG): container finished" podID="3da8a195-2855-42e0-8035-e8991fd5b0fd" containerID="f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df" exitCode=0
Nov 04 00:28:26.112957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.112778    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"
Nov 04 00:28:26.112957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.112803    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5" event={"ID":"3da8a195-2855-42e0-8035-e8991fd5b0fd","Type":"ContainerDied","Data":"f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df"}
Nov 04 00:28:26.112957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.112865    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5" event={"ID":"3da8a195-2855-42e0-8035-e8991fd5b0fd","Type":"ContainerDied","Data":"669742f147b06feddc01954b91cfb70acacc612fd49fb831c1c82899be336121"}
Nov 04 00:28:26.112957 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.112880    5074 scope.go:122] "RemoveContainer" containerID="f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df"
Nov 04 00:28:26.129491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.129463    5074 scope.go:122] "RemoveContainer" containerID="f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df"
Nov 04 00:28:26.130246 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:26.130216    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df\": container with ID starting with f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df not found: ID does not exist" containerID="f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df"
Nov 04 00:28:26.130392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.130257    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df"} err="failed to get container status \"f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df\": rpc error: code = NotFound desc = could not find container \"f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df\": container with ID starting with f9464d8d1b9c1ae7bc88438b2977d6bd23b1401f168e8e7e66c667bc13cab0df not found: ID does not exist"
Nov 04 00:28:26.149785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.149721    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"]
Nov 04 00:28:26.153939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:26.153914    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6382fc-75db8977bb-2vbx5"]
Nov 04 00:28:27.037270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:27.037234    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da8a195-2855-42e0-8035-e8991fd5b0fd" path="/var/lib/kubelet/pods/3da8a195-2855-42e0-8035-e8991fd5b0fd/volumes"
Nov 04 00:28:28.030422 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:28.030376    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:28:31.351361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:31.350112    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"]
Nov 04 00:28:31.352346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:31.352314    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api" containerID="cri-o://806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af" gracePeriod=30
Nov 04 00:28:31.983239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:31.980548    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:28:32.016066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.009608    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs\") pod \"50149425-2487-4c92-b0e1-9790ef804e3e\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") "
Nov 04 00:28:32.016066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.009730    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca\") pod \"50149425-2487-4c92-b0e1-9790ef804e3e\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") "
Nov 04 00:28:32.016066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.009864    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls\") pod \"50149425-2487-4c92-b0e1-9790ef804e3e\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") "
Nov 04 00:28:32.016066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.009919    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b\" (UniqueName: \"kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b\") pod \"50149425-2487-4c92-b0e1-9790ef804e3e\" (UID: \"50149425-2487-4c92-b0e1-9790ef804e3e\") "
Nov 04 00:28:32.019763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.018632    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca" pod "50149425-2487-4c92-b0e1-9790ef804e3e" (UID: "50149425-2487-4c92-b0e1-9790ef804e3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:28:32.028760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.028294    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b" pod "50149425-2487-4c92-b0e1-9790ef804e3e" (UID: "50149425-2487-4c92-b0e1-9790ef804e3e"). InnerVolumeSpecName "kube-api-access-ggn2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:28:32.032770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.032260    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls" pod "50149425-2487-4c92-b0e1-9790ef804e3e" (UID: "50149425-2487-4c92-b0e1-9790ef804e3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:28:32.036759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.034527    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs" pod "50149425-2487-4c92-b0e1-9790ef804e3e" (UID: "50149425-2487-4c92-b0e1-9790ef804e3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:28:32.115787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.115720    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-maas-api-tls\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:32.115787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.115781    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggn2b\" (UniqueName: \"kubernetes.io/projected/50149425-2487-4c92-b0e1-9790ef804e3e-kube-api-access-ggn2b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:32.116131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.115799    5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50149425-2487-4c92-b0e1-9790ef804e3e-metrics-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:32.116131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.115816    5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50149425-2487-4c92-b0e1-9790ef804e3e-openshift-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:32.187192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.187151    5074 generic.go:356] "Generic (PLEG): container finished" podID="50149425-2487-4c92-b0e1-9790ef804e3e" containerID="806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af" exitCode=0
Nov 04 00:28:32.187559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.187539    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" event={"ID":"50149425-2487-4c92-b0e1-9790ef804e3e","Type":"ContainerDied","Data":"806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af"}
Nov 04 00:28:32.187730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.187715    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k" event={"ID":"50149425-2487-4c92-b0e1-9790ef804e3e","Type":"ContainerDied","Data":"6bffd4f64ae8dea3650e7f5b3d12ae3c3b4a19ad3f854aa6ba4af751c78b5465"}
Nov 04 00:28:32.187879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.187865    5074 scope.go:122] "RemoveContainer" containerID="806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af"
Nov 04 00:28:32.188144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.188128    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"
Nov 04 00:28:32.218928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.218876    5074 scope.go:122] "RemoveContainer" containerID="f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"
Nov 04 00:28:32.233322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.233294    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"]
Nov 04 00:28:32.247808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.247759    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6382fc-559588688-hjt6k"]
Nov 04 00:28:32.248535 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.248510    5074 scope.go:122] "RemoveContainer" containerID="806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af"
Nov 04 00:28:32.249520 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:32.249491    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af\": container with ID starting with 806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af not found: ID does not exist" containerID="806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af"
Nov 04 00:28:32.249671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.249534    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af"} err="failed to get container status \"806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af\": rpc error: code = NotFound desc = could not find container \"806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af\": container with ID starting with 806d187e9853575f1702de565e32b115629d789cda02991d77b6b3200a33a6af not found: ID does not exist"
Nov 04 00:28:32.249671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.249560    5074 scope.go:122] "RemoveContainer" containerID="f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"
Nov 04 00:28:32.250048 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:32.249874    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a\": container with ID starting with f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a not found: ID does not exist" containerID="f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"
Nov 04 00:28:32.250048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:32.249903    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a"} err="failed to get container status \"f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a\": rpc error: code = NotFound desc = could not find container \"f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a\": container with ID starting with f0219353f1061b9216e0aee890fcb07a96ba7d0519afeb2960e81bfc899db64a not found: ID does not exist"
Nov 04 00:28:33.086763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:33.086409    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" path="/var/lib/kubelet/pods/50149425-2487-4c92-b0e1-9790ef804e3e/volumes"
Nov 04 00:28:37.684702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:37.684663    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"]
Nov 04 00:28:37.685133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:37.684994    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy" containerID="cri-o://8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de" gracePeriod=30
Nov 04 00:28:43.006886 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.006853    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:28:43.142119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142079    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142165    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142225    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142261    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142340    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142437    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142517    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:28:43.142700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142535    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142536    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:28:43.142700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142574    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142643    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo\") pod \"113595e4-c554-4735-8d85-6e719e9b8c24\" (UID: \"113595e4-c554-4735-8d85-6e719e9b8c24\") "
Nov 04 00:28:43.142951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142805    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:28:43.142951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.142815    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:28:43.143362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.143336    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.143513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.143369    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.143513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.143387    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.143513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.143403    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.143513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.143448    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:28:43.144980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.144948    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:28:43.145142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.145119    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:28:43.145579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.145551    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:28:43.145697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.145676    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78" pod "113595e4-c554-4735-8d85-6e719e9b8c24" (UID: "113595e4-c554-4735-8d85-6e719e9b8c24"). InnerVolumeSpecName "kube-api-access-56t78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:28:43.244778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.244664    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56t78\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-kube-api-access-56t78\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.244778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.244701    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113595e4-c554-4735-8d85-6e719e9b8c24-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.244778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.244712    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113595e4-c554-4735-8d85-6e719e9b8c24-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.244778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.244721    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113595e4-c554-4735-8d85-6e719e9b8c24-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.244778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.244730    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113595e4-c554-4735-8d85-6e719e9b8c24-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:28:43.348346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.348304    5074 generic.go:356] "Generic (PLEG): container finished" podID="113595e4-c554-4735-8d85-6e719e9b8c24" containerID="8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de" exitCode=0
Nov 04 00:28:43.348527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.348352    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" event={"ID":"113595e4-c554-4735-8d85-6e719e9b8c24","Type":"ContainerDied","Data":"8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de"}
Nov 04 00:28:43.348527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.348374    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"
Nov 04 00:28:43.348527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.348416    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w" event={"ID":"113595e4-c554-4735-8d85-6e719e9b8c24","Type":"ContainerDied","Data":"0e43cf7fe38d1f751e404f8f09d7ac9710e57dfbcb74937319d81c291217a1bf"}
Nov 04 00:28:43.348527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.348443    5074 scope.go:122] "RemoveContainer" containerID="8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de"
Nov 04 00:28:43.362299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.362278    5074 scope.go:122] "RemoveContainer" containerID="b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"
Nov 04 00:28:43.373110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.373084    5074 scope.go:122] "RemoveContainer" containerID="8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de"
Nov 04 00:28:43.373404 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:43.373381    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de\": container with ID starting with 8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de not found: ID does not exist" containerID="8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de"
Nov 04 00:28:43.373489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.373412    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de"} err="failed to get container status \"8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de\": rpc error: code = NotFound desc = could not find container \"8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de\": container with ID starting with 8ebd083a15f24d3b610470cd783779d97252415ecfbb52d30095ed13705381de not found: ID does not exist"
Nov 04 00:28:43.373489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.373430    5074 scope.go:122] "RemoveContainer" containerID="b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"
Nov 04 00:28:43.373811 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:43.373724    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b\": container with ID starting with b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b not found: ID does not exist" containerID="b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"
Nov 04 00:28:43.373969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.373949    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b"} err="failed to get container status \"b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b\": rpc error: code = NotFound desc = could not find container \"b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b\": container with ID starting with b909668c0e8eae12662e49ad0c4e1701b0385f2a1edfb1b7d746f8eab0a3182b not found: ID does not exist"
Nov 04 00:28:43.377888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.377860    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"]
Nov 04 00:28:43.382330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:43.382310    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6382fc-openshift-default-646f49ff7-5rv6w"]
Nov 04 00:28:45.021806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:45.021768    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" path="/var/lib/kubelet/pods/113595e4-c554-4735-8d85-6e719e9b8c24/volumes"
Nov 04 00:28:55.958935 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:55.958885    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:28:55.973385 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:28:55.973330    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:28:58.025582 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:28:58.025539    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:29:00.844533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:00.844498    5074 scope.go:122] "RemoveContainer" containerID="b35da35267f699b70f845681fa173040a975ffa540336869636e3d9d826d220f"
Nov 04 00:29:00.859007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:00.858957    5074 scope.go:122] "RemoveContainer" containerID="595065dbdf146fef549c9299edd51a4df9f5f7dc4ca6524496ce24a5e2945301"
Nov 04 00:29:26.229487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.229449    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"]
Nov 04 00:29:26.230270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230252    5074 cpu_manager.go:414] "containerMap: removing container" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api"
Nov 04 00:29:26.230336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230270    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api"
Nov 04 00:29:26.230336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230292    5074 cpu_manager.go:414] "containerMap: removing container" podUID="36fce481-441d-4b3f-93e3-67646d320b00" containerName="revoke-keys"
Nov 04 00:29:26.230336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230297    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fce481-441d-4b3f-93e3-67646d320b00" containerName="revoke-keys"
Nov 04 00:29:26.230336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230316    5074 cpu_manager.go:414] "containerMap: removing container" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy"
Nov 04 00:29:26.230336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230321    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230339    5074 cpu_manager.go:414] "containerMap: removing container" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230347    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230371    5074 cpu_manager.go:414] "containerMap: removing container" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerName="payload-pre-processing"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230378    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerName="payload-pre-processing"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230389    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerName="payload-processing"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230396    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerName="payload-processing"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230406    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3da8a195-2855-42e0-8035-e8991fd5b0fd" containerName="payload-processing"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230416    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da8a195-2855-42e0-8035-e8991fd5b0fd" containerName="payload-processing"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230429    5074 cpu_manager.go:414] "containerMap: removing container" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api"
Nov 04 00:29:26.230510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230438    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230603    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="12115daa-a907-406d-ba0d-1b8b664d24ef" containerName="payload-pre-processing"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230613    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8b2a7f-a0b7-43d4-b917-20f3d4f0062d" containerName="payload-processing"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230621    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da8a195-2855-42e0-8035-e8991fd5b0fd" containerName="payload-processing"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230629    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230638    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="50149425-2487-4c92-b0e1-9790ef804e3e" containerName="maas-api"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230647    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fce481-441d-4b3f-93e3-67646d320b00" containerName="revoke-keys"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230656    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy"
Nov 04 00:29:26.230846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.230663    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="113595e4-c554-4735-8d85-6e719e9b8c24" containerName="istio-proxy"
Nov 04 00:29:26.236818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.236378    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.244129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.244100    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:29:26.244384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.244362    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:29:26.248019 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.247996    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"]
Nov 04 00:29:26.368295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.368264    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqzgs\" (UniqueName: \"kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.368462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.368401    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.368515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.368483    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.469397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.469362    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.469561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.469437    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.469610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.469563    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqzgs\" (UniqueName: \"kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.470183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.470151    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.471856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.471833    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.482763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.482701    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqzgs\" (UniqueName: \"kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs\") pod \"maas-api-revoke-sub-cb2a1248-jnn5s\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.560763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.560707    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:26.850310 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:29:26.850278    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbd11a0_78cb_4572_a6f7_d3010494bbc6.slice/crio-a9b83c81250cd8c89259f4d5797cc9510851d315bc656b65224d9e2d2c93cfa2 WatchSource:0}: Error finding container a9b83c81250cd8c89259f4d5797cc9510851d315bc656b65224d9e2d2c93cfa2: Status 404 returned error can't find the container with id a9b83c81250cd8c89259f4d5797cc9510851d315bc656b65224d9e2d2c93cfa2
Nov 04 00:29:26.851176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:26.851151    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"]
Nov 04 00:29:27.039562 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:27.039519    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"]
Nov 04 00:29:27.043494 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:27.043462    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b4a1c9f-dhzsw"]
Nov 04 00:29:27.771969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:27.771928    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s" event={"ID":"adbd11a0-78cb-4572-a6f7-d3010494bbc6","Type":"ContainerStarted","Data":"561aa0637aa6b10173d145abda447f8514b1287fa7a5059c30cc805e43409d30"}
Nov 04 00:29:27.772435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:27.771996    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s" event={"ID":"adbd11a0-78cb-4572-a6f7-d3010494bbc6","Type":"ContainerStarted","Data":"a9b83c81250cd8c89259f4d5797cc9510851d315bc656b65224d9e2d2c93cfa2"}
Nov 04 00:29:27.794504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:27.794450    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s" podStartSLOduration=1.794436241 podStartE2EDuration="1.794436241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:29:27.790192181 +0000 UTC m=+3632.889792327" watchObservedRunningTime="2026-11-04 00:29:27.794436241 +0000 UTC m=+3632.894036385"
Nov 04 00:29:28.027363 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:29:28.027292    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:29:28.781133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:28.781099    5074 generic.go:356] "Generic (PLEG): container finished" podID="adbd11a0-78cb-4572-a6f7-d3010494bbc6" containerID="561aa0637aa6b10173d145abda447f8514b1287fa7a5059c30cc805e43409d30" exitCode=0
Nov 04 00:29:28.781539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:28.781189    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s" event={"ID":"adbd11a0-78cb-4572-a6f7-d3010494bbc6","Type":"ContainerDied","Data":"561aa0637aa6b10173d145abda447f8514b1287fa7a5059c30cc805e43409d30"}
Nov 04 00:29:29.020332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:29.020295    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9077fc41-c392-456d-960c-07bed701a219" path="/var/lib/kubelet/pods/9077fc41-c392-456d-960c-07bed701a219/volumes"
Nov 04 00:29:29.041273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:29.041187    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"]
Nov 04 00:29:29.047548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:29.047515    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e58553-m4g5h"]
Nov 04 00:29:29.986143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:29.986112    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:30.013449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.013290    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs\" (UniqueName: \"kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs\") pod \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") "
Nov 04 00:29:30.013449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.013357    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca\") pod \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") "
Nov 04 00:29:30.013770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.013494    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca\") pod \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\" (UID: \"adbd11a0-78cb-4572-a6f7-d3010494bbc6\") "
Nov 04 00:29:30.014398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.014296    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca" pod "adbd11a0-78cb-4572-a6f7-d3010494bbc6" (UID: "adbd11a0-78cb-4572-a6f7-d3010494bbc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:29:30.018892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.017169    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca" pod "adbd11a0-78cb-4572-a6f7-d3010494bbc6" (UID: "adbd11a0-78cb-4572-a6f7-d3010494bbc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:29:30.019015 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.018921    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs" pod "adbd11a0-78cb-4572-a6f7-d3010494bbc6" (UID: "adbd11a0-78cb-4572-a6f7-d3010494bbc6"). InnerVolumeSpecName "kube-api-access-dqzgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:29:30.115890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.115790    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqzgs\" (UniqueName: \"kubernetes.io/projected/adbd11a0-78cb-4572-a6f7-d3010494bbc6-kube-api-access-dqzgs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:29:30.115890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.115832    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:29:30.115890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.115847    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbd11a0-78cb-4572-a6f7-d3010494bbc6-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:29:30.799583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.799548    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"
Nov 04 00:29:30.799583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.799570    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s" event={"ID":"adbd11a0-78cb-4572-a6f7-d3010494bbc6","Type":"ContainerDied","Data":"a9b83c81250cd8c89259f4d5797cc9510851d315bc656b65224d9e2d2c93cfa2"}
Nov 04 00:29:30.799856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:30.799630    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b83c81250cd8c89259f4d5797cc9510851d315bc656b65224d9e2d2c93cfa2"
Nov 04 00:29:31.022987 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:31.022951    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cb039f-65fb-49f8-b51e-18591725dcfe" path="/var/lib/kubelet/pods/54cb039f-65fb-49f8-b51e-18591725dcfe/volumes"
Nov 04 00:29:38.037944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:38.037854    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"]
Nov 04 00:29:38.041795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:38.041761    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e20d50-2gddm"]
Nov 04 00:29:39.020572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:39.020538    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85a8d8f-62e4-42d3-a495-ae8f7e04625a" path="/var/lib/kubelet/pods/e85a8d8f-62e4-42d3-a495-ae8f7e04625a/volumes"
Nov 04 00:29:43.034466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:43.034415    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"]
Nov 04 00:29:43.038853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:43.038822    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33619f16-5cx82"]
Nov 04 00:29:45.021181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:29:45.021149    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396e8e57-5200-4abf-a4f5-23dc5152f440" path="/var/lib/kubelet/pods/396e8e57-5200-4abf-a4f5-23dc5152f440/volumes"
Nov 04 00:29:58.030598 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:29:58.030555    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:30:00.150850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.150816    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"]
Nov 04 00:30:00.151586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.151570    5074 cpu_manager.go:414] "containerMap: removing container" podUID="adbd11a0-78cb-4572-a6f7-d3010494bbc6" containerName="revoke-keys"
Nov 04 00:30:00.151635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.151588    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbd11a0-78cb-4572-a6f7-d3010494bbc6" containerName="revoke-keys"
Nov 04 00:30:00.151731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.151722    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbd11a0-78cb-4572-a6f7-d3010494bbc6" containerName="revoke-keys"
Nov 04 00:30:00.155405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.155383    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"]
Nov 04 00:30:00.155530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.155500    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.159888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.159864    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"]
Nov 04 00:30:00.160000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.159954    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.165858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.165832    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:30:00.172917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.172891    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"]
Nov 04 00:30:00.174443 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.173102    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.179490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.179465    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"]
Nov 04 00:30:00.179806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.179781    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.187726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.187479    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"]
Nov 04 00:30:00.187726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.187525    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"]
Nov 04 00:30:00.187726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.187552    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"]
Nov 04 00:30:00.187726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.187640    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.193291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.193263    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"]
Nov 04 00:30:00.198657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.198633    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"]
Nov 04 00:30:00.222758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.212227    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"]
Nov 04 00:30:00.232021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.231989    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.234942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.234907    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"]
Nov 04 00:30:00.269291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.265804    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.277066 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.277029    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"]
Nov 04 00:30:00.285946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.285920    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"]
Nov 04 00:30:00.286851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.286402    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.286851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.286522    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.286851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.286555    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjrn\" (UniqueName: \"kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.286851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.286676    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48x5c\" (UniqueName: \"kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.286851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.286781    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blzp\" (UniqueName: \"kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp\") pod \"maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.287814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287474    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.287814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287548    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.287814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287644    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.287814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287671    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.287814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287721    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fj9\" (UniqueName: \"kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.287814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287805    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cf2x\" (UniqueName: \"kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.288204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.287856    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wkrv\" (UniqueName: \"kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.288362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.286069    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.297061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.297036    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"]
Nov 04 00:30:00.301428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.301405    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.305840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.305820    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.308084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.308066    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"]
Nov 04 00:30:00.313924 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.313900    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"]
Nov 04 00:30:00.319350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.319329    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"]
Nov 04 00:30:00.322144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.322122    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"]
Nov 04 00:30:00.327802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.327783    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"]
Nov 04 00:30:00.344944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.344913    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"]
Nov 04 00:30:00.345456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.345435    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.354218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.354194    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"]
Nov 04 00:30:00.355118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.354994    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.368801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.366835    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"]
Nov 04 00:30:00.368801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.366888    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"]
Nov 04 00:30:00.368801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.366925    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"]
Nov 04 00:30:00.368801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.367022    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.372601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.372576    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"]
Nov 04 00:30:00.379903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.379879    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"]
Nov 04 00:30:00.386919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.386896    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"]
Nov 04 00:30:00.387063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.387002    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.389420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389271    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzdk\" (UniqueName: \"kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.389420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389311    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78pj\" (UniqueName: \"kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.389420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389403    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.389640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389468    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.389640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389538    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.389640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389585    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.389965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389648    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.389965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389677    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjrn\" (UniqueName: \"kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.389965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389718    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkc5\" (UniqueName: \"kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.389965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389872    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.389965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389904    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.389965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.389947    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48x5c\" (UniqueName: \"kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390034    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blzp\" (UniqueName: \"kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp\") pod \"maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390076    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390110    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390144    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2n9c\" (UniqueName: \"kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390192    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdbh\" (UniqueName: \"kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390236    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390263    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.390361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390299    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fj9\" (UniqueName: \"kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.390818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390371    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cf2x\" (UniqueName: \"kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.390818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390408    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wkrv\" (UniqueName: \"kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.390818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.390513    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.391555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.391419    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.391555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.391419    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.392384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.391832    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.392384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.391994    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.392384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.392342    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.394784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.393963    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"]
Nov 04 00:30:00.394784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.394060    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.398641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.398619    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.405407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.405105    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wkrv\" (UniqueName: \"kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv\") pod \"maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.405643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.405550    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blzp\" (UniqueName: \"kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp\") pod \"maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.405643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.405619    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"]
Nov 04 00:30:00.406391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.406348    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjrn\" (UniqueName: \"kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn\") pod \"maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.407792 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.407118    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fj9\" (UniqueName: \"kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.409227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.409201    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cf2x\" (UniqueName: \"kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.409954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.409924    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48x5c\" (UniqueName: \"kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.410577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.410555    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"]
Nov 04 00:30:00.413692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.413670    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"]
Nov 04 00:30:00.417219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.417198    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"]
Nov 04 00:30:00.425326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.425305    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"]
Nov 04 00:30:00.426362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.426341    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.432725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.432702    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"]
Nov 04 00:30:00.432881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.432818    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.436818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.436571    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"]
Nov 04 00:30:00.481164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.481132    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:00.491546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491511    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.491685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491564    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vr7\" (UniqueName: \"kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.491685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491621    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkc5\" (UniqueName: \"kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.491685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491676    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.491907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491706    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.491907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491732    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.491907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491795    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7pp\" (UniqueName: \"kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.491907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491838    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w6v\" (UniqueName: \"kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.491907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491891    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.492123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491922    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.492123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491957    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjjs\" (UniqueName: \"kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.492123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.491987    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.492123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492023    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2n9c\" (UniqueName: \"kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.492123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492052    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdbh\" (UniqueName: \"kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492146    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tz4m\" (UniqueName: \"kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492181    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lk6\" (UniqueName: \"kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492213    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzdk\" (UniqueName: \"kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492237    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78pj\" (UniqueName: \"kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492274    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492304    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.492342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492337    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.492775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492404    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkn48\" (UniqueName: \"kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.492775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492430    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.492775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.492462    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.493297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.493268    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.494365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.494334    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.494509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.494481    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.494595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.494564    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.495037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.495013    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.509038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.509000    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkc5\" (UniqueName: \"kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.510458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.510423    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzdk\" (UniqueName: \"kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.510571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.510541    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2n9c\" (UniqueName: \"kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.511203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.511180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78pj\" (UniqueName: \"kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj\") pod \"maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.512171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.512146    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdbh\" (UniqueName: \"kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.519808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.519782    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:00.544047 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.543728    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:00.577751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.577704    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.593880    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tz4m\" (UniqueName: \"kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.593938    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lk6\" (UniqueName: \"kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.593986    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594065    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkn48\" (UniqueName: \"kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594109    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594188    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594216    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vr7\" (UniqueName: \"kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594317    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594366    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7pp\" (UniqueName: \"kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594420    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w6v\" (UniqueName: \"kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594477    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594509    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594548    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjjs\" (UniqueName: \"kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.595789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.594577    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.614240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.597899    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.614240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.602712    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.614240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.603466    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.614240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.612315    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.614240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.613485    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.635716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.622907    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.635716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.635495    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:00.636888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.636208    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.641022    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.645689    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w6v\" (UniqueName: \"kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.649275    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjjs\" (UniqueName: \"kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.649494    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.651933    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.652510    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lk6\" (UniqueName: \"kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.653206    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vr7\" (UniqueName: \"kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.653431    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tz4m\" (UniqueName: \"kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:00.654106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.653484    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:00.656205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.655659    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkn48\" (UniqueName: \"kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.656205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.656160    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7pp\" (UniqueName: \"kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp\") pod \"maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.677728 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.677465    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:00.688296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.688263    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:00.700182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.699978    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:00.765636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.765531    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:00.773981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.773945    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:00.783145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.782954    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:00.819202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.812701    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:00.844786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.844498    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:00.882867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:00.882756    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:01.093051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.089692    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"]
Nov 04 00:30:01.165727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.165601    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerStarted","Data":"c2b3f32c07b9ce0cc18f603e51efb1a446241b1e82d6d6b62e51e519b810b6be"}
Nov 04 00:30:01.280588 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.280557    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a6425b_0808_4542_8c65_d16448a10baf.slice/crio-98cc20b9edb36923a0c5f2e1761e528538368eca48e9c02b48f145a74030d8fd WatchSource:0}: Error finding container 98cc20b9edb36923a0c5f2e1761e528538368eca48e9c02b48f145a74030d8fd: Status 404 returned error can't find the container with id 98cc20b9edb36923a0c5f2e1761e528538368eca48e9c02b48f145a74030d8fd
Nov 04 00:30:01.283248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.283218    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"]
Nov 04 00:30:01.444591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.444541    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"]
Nov 04 00:30:01.494345 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.494308    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"]
Nov 04 00:30:01.510347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.509502    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"]
Nov 04 00:30:01.569682 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.564021    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96a4d5e_43f2_4bbf_a3bb_483a0ab53ad3.slice/crio-2ad117073f264381fa18cc403ce87cb054bb0022afa2bde8bf80e94edd2eb0d1 WatchSource:0}: Error finding container 2ad117073f264381fa18cc403ce87cb054bb0022afa2bde8bf80e94edd2eb0d1: Status 404 returned error can't find the container with id 2ad117073f264381fa18cc403ce87cb054bb0022afa2bde8bf80e94edd2eb0d1
Nov 04 00:30:01.569682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.566621    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"]
Nov 04 00:30:01.579877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.579845    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"]
Nov 04 00:30:01.596893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.596862    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"]
Nov 04 00:30:01.639239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.639205    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"]
Nov 04 00:30:01.640050 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.640019    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4961c11_d4e7_4ccd_99e8_13766e5fea74.slice/crio-1fd3c6d58fda6f237c1f937b5c48209d47fee15092b34f5e48aced4e163a9a1a WatchSource:0}: Error finding container 1fd3c6d58fda6f237c1f937b5c48209d47fee15092b34f5e48aced4e163a9a1a: Status 404 returned error can't find the container with id 1fd3c6d58fda6f237c1f937b5c48209d47fee15092b34f5e48aced4e163a9a1a
Nov 04 00:30:01.653859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.653826    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"]
Nov 04 00:30:01.662707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.662672    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"]
Nov 04 00:30:01.675874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.675520    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"]
Nov 04 00:30:01.676823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.676783    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"]
Nov 04 00:30:01.683641 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.683584    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d407d9a_345c_495f_9ab2_2f2eb28bf209.slice/crio-ff6a0ea2748480a7219591add79ebc442909fdbfd9bea29973cd3fe9cf280e1b WatchSource:0}: Error finding container ff6a0ea2748480a7219591add79ebc442909fdbfd9bea29973cd3fe9cf280e1b: Status 404 returned error can't find the container with id ff6a0ea2748480a7219591add79ebc442909fdbfd9bea29973cd3fe9cf280e1b
Nov 04 00:30:01.766113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.766075    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"]
Nov 04 00:30:01.777663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.777632    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"]
Nov 04 00:30:01.785555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.785532    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"]
Nov 04 00:30:01.789563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.789539    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"]
Nov 04 00:30:01.808799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:01.808774    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"]
Nov 04 00:30:01.925010 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.924980    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590353e1_672c_467b_8a8d_ecc4cf08bb55.slice/crio-1daf59f0bb1b4386d66d6ff992e26cf10066c0da61e8eab2323b6f9b348c44ad WatchSource:0}: Error finding container 1daf59f0bb1b4386d66d6ff992e26cf10066c0da61e8eab2323b6f9b348c44ad: Status 404 returned error can't find the container with id 1daf59f0bb1b4386d66d6ff992e26cf10066c0da61e8eab2323b6f9b348c44ad
Nov 04 00:30:01.927599 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.927565    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b8bc1c_80fd_4dda_b44c_6996cc16fe12.slice/crio-1dd60fbae43b950e4259adda23ba8645e846ed49781d5093f310c2aa3a2535e4 WatchSource:0}: Error finding container 1dd60fbae43b950e4259adda23ba8645e846ed49781d5093f310c2aa3a2535e4: Status 404 returned error can't find the container with id 1dd60fbae43b950e4259adda23ba8645e846ed49781d5093f310c2aa3a2535e4
Nov 04 00:30:01.940998 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:01.940970    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484c335d_2c01_4c31_a93a_528c2361fb96.slice/crio-6fc6031abbfc8c69127674aa757c92d5ac5df19729c414024d0e7014623af0b6 WatchSource:0}: Error finding container 6fc6031abbfc8c69127674aa757c92d5ac5df19729c414024d0e7014623af0b6: Status 404 returned error can't find the container with id 6fc6031abbfc8c69127674aa757c92d5ac5df19729c414024d0e7014623af0b6
Nov 04 00:30:02.177835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.177787    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerStarted","Data":"93e42579522b9eb648deb1913c87c0ac9e2cadb9a36f72d640b58080693d183a"}
Nov 04 00:30:02.179441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.179403    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerStarted","Data":"5efad237e954f9ca354db3b30a133d9cb19bb613c6701bc1f5acf9c8206be442"}
Nov 04 00:30:02.181000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.180970    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerStarted","Data":"1dd60fbae43b950e4259adda23ba8645e846ed49781d5093f310c2aa3a2535e4"}
Nov 04 00:30:02.182434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.182410    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerStarted","Data":"680c1849cf43c862724635210f46bf34c4bfcf524e4e5d7f41988da4f6d5f43b"}
Nov 04 00:30:02.183861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.183831    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerStarted","Data":"2ad117073f264381fa18cc403ce87cb054bb0022afa2bde8bf80e94edd2eb0d1"}
Nov 04 00:30:02.185188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.185154    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerStarted","Data":"de6d6a8d9542ac773c5b6521af2c86e5d6c4d93c3515aeb7582a311c00480be1"}
Nov 04 00:30:02.186461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.186436    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerStarted","Data":"6fc6031abbfc8c69127674aa757c92d5ac5df19729c414024d0e7014623af0b6"}
Nov 04 00:30:02.187703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.187682    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" event={"ID":"590353e1-672c-467b-8a8d-ecc4cf08bb55","Type":"ContainerStarted","Data":"1daf59f0bb1b4386d66d6ff992e26cf10066c0da61e8eab2323b6f9b348c44ad"}
Nov 04 00:30:02.189292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.189267    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerStarted","Data":"9858583eb8ca37e8a4b1ecee0984add60dd4231ddf68e48677971f38a18b71ff"}
Nov 04 00:30:02.189402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.189316    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerStarted","Data":"98cc20b9edb36923a0c5f2e1761e528538368eca48e9c02b48f145a74030d8fd"}
Nov 04 00:30:02.190868 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.190846    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerStarted","Data":"ff6a0ea2748480a7219591add79ebc442909fdbfd9bea29973cd3fe9cf280e1b"}
Nov 04 00:30:02.192600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.192576    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" event={"ID":"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e","Type":"ContainerStarted","Data":"78e2a53aeb761976ea99c0b25aef83f89ab18cba0383cf8a1b6e55ccc05fa810"}
Nov 04 00:30:02.194036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.194012    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerStarted","Data":"1fd3c6d58fda6f237c1f937b5c48209d47fee15092b34f5e48aced4e163a9a1a"}
Nov 04 00:30:02.195354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.195319    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerStarted","Data":"0217bc2ebef457e1f0541936b20581b6459cf6ab5fc2fabdef5388fadfebae63"}
Nov 04 00:30:02.196929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.196853    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerStarted","Data":"340aa3d3807382a66b3bc6009d52466d506b1a9ed23055aedbe2e1cacea0db25"}
Nov 04 00:30:02.198094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.198048    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerStarted","Data":"45ceedb4bd91f5a8c9bfb48625963b4a95e0a5434291453be345504ebc66dc83"}
Nov 04 00:30:02.201157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.201104    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerStarted","Data":"749b3675fed3622ee493129901318d24883a8f9612d92d7918e7ee60fd709076"}
Nov 04 00:30:02.204390 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.204312    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerStarted","Data":"53d11a7809c6f5dded66454b297a22d9cb84dc39110295cd2527cc87462161ec"}
Nov 04 00:30:02.206275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.206207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" event={"ID":"24b7122f-745a-4828-b1c1-17a3ac6baa04","Type":"ContainerStarted","Data":"21bfc0ca777e9e30ed74e3cd8cc1f7da55dbadc34753b40b13e4b085a3e202f8"}
Nov 04 00:30:02.229417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.229366    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" podStartSLOduration=2.229353348 podStartE2EDuration="2.229353348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:02.226411149 +0000 UTC m=+3667.326011296" watchObservedRunningTime="2026-11-04 00:30:02.229353348 +0000 UTC m=+3667.328953520"
Nov 04 00:30:02.250786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:02.250702    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" podStartSLOduration=2.250684149 podStartE2EDuration="2.250684149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:02.244383695 +0000 UTC m=+3667.343983840" watchObservedRunningTime="2026-11-04 00:30:02.250684149 +0000 UTC m=+3667.350284295"
Nov 04 00:30:03.051097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.051069    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"]
Nov 04 00:30:03.067035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.067005    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"]
Nov 04 00:30:03.067182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.067113    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.072710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.072656    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"sub-test-model-b3bea7-kserve-self-signed-certs\""
Nov 04 00:30:03.072994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.072974    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"default-dockercfg-xb2f8\""
Nov 04 00:30:03.074690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.073000    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"kube-root-ca.crt\""
Nov 04 00:30:03.074690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.072889    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b3bea7\"/\"openshift-service-ca.crt\""
Nov 04 00:30:03.187597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.187557    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9v9r\" (UniqueName: \"kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.188270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.188242    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.188394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.188291    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.188588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.188390    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.188677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.188648    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.188813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.188797    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.232851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.231641    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerStarted","Data":"1557e62413e2bbb40aeee3f20f3436bba8d8a6fe1204a543cebf7b63e2210c47"}
Nov 04 00:30:03.235379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.235344    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerStarted","Data":"8d4755236d9afe0e0a292f34bf71c91539fd78d5ad88c1401240a210496d34e5"}
Nov 04 00:30:03.238762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.238719    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerStarted","Data":"b5a298e448b0e1dc55bb2e017dd48c78bd92ca0100422e6749b1b8b2cf06cb4d"}
Nov 04 00:30:03.241852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.241827    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerStarted","Data":"4e4ef006ed6738351ee3c13feb35e247dccd8cb9b8b4197f077d145ed20462a5"}
Nov 04 00:30:03.245435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.245410    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerStarted","Data":"6ac09aaa45c9c06230eb8d21905822eadf84ea1fde792f03bcdb4453f1c33769"}
Nov 04 00:30:03.247888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.247863    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerStarted","Data":"7a51956e1c212f468083318310fad88d0772f6468dbcea717d0fbbb839923f71"}
Nov 04 00:30:03.250879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.250855    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerStarted","Data":"88ccd7a777ad6d6a58ba795d5fdf003a35173e476572ad8f545f0ad8d86fdb06"}
Nov 04 00:30:03.254087 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.254062    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerStarted","Data":"c65736aa7e0525135cd0170f7dfed42a374504a2234fa36475f90b8d26b34b91"}
Nov 04 00:30:03.257809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.257785    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerStarted","Data":"97e92372680c0d288506fa591bed4ee1d6ce7c0129e4f46b120fe1c8b22b0bbe"}
Nov 04 00:30:03.261851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.261827    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerStarted","Data":"9468956e4083707069cf5d454cc6ed9eae89307ab2ddc5b84adc9e1d8e337491"}
Nov 04 00:30:03.271080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.270425    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" event={"ID":"590353e1-672c-467b-8a8d-ecc4cf08bb55","Type":"ContainerStarted","Data":"4c310a680ebd4d614d37efe26ff00d2117aa2ff8803dbf1a9717be9bb770f08d"}
Nov 04 00:30:03.272501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.272442    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" podStartSLOduration=3.2724272660000002 podStartE2EDuration="3.272427266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.259424475 +0000 UTC m=+3668.359024621" watchObservedRunningTime="2026-11-04 00:30:03.272427266 +0000 UTC m=+3668.372027413"
Nov 04 00:30:03.280962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.280934    5074 generic.go:356] "Generic (PLEG): container finished" podID="11a6425b-0808-4542-8c65-d16448a10baf" containerID="9858583eb8ca37e8a4b1ecee0984add60dd4231ddf68e48677971f38a18b71ff" exitCode=60
Nov 04 00:30:03.365255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.281429    5074 scope.go:122] "RemoveContainer" containerID="9858583eb8ca37e8a4b1ecee0984add60dd4231ddf68e48677971f38a18b71ff"
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.308772    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.368135    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.368444    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9v9r\" (UniqueName: \"kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.368543    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.344031    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.317099    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerDied","Data":"9858583eb8ca37e8a4b1ecee0984add60dd4231ddf68e48677971f38a18b71ff"}
Nov 04 00:30:03.374414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.369784    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.385682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.382248    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.385682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.368597    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.385682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.383788    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.385682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.384443    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.385682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.385443    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" podStartSLOduration=3.385427413 podStartE2EDuration="3.385427413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.281732917 +0000 UTC m=+3668.381333061" watchObservedRunningTime="2026-11-04 00:30:03.385427413 +0000 UTC m=+3668.485027558"
Nov 04 00:30:03.389353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.388954    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.409599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.409542    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerStarted","Data":"efbea6a6ce827cbb0752bae6300bc93a4ff3e0599bc243ff9ab79222d864e577"}
Nov 04 00:30:03.446867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.424447    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9v9r\" (UniqueName: \"kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r\") pod \"sub-test-model-b3bea7-kserve-6b5949bf97-px7pr\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") " pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.446867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.432360    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" event={"ID":"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e","Type":"ContainerStarted","Data":"58294aca6587648da14700e37967df8657b425deccd5a638ad6b4bd45071b8a8"}
Nov 04 00:30:03.446867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.433627    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" podStartSLOduration=3.433609929 podStartE2EDuration="3.433609929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.412263736 +0000 UTC m=+3668.511863880" watchObservedRunningTime="2026-11-04 00:30:03.433609929 +0000 UTC m=+3668.533210072"
Nov 04 00:30:03.446867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.442472    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" podStartSLOduration=3.442439357 podStartE2EDuration="3.442439357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.438183976 +0000 UTC m=+3668.537784121" watchObservedRunningTime="2026-11-04 00:30:03.442439357 +0000 UTC m=+3668.542039502"
Nov 04 00:30:03.463961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.450291    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerStarted","Data":"9661ffe763edc6dd1d2720da56402f3765269945263f998717d6c969085d888b"}
Nov 04 00:30:03.463961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.455730    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerStarted","Data":"4f0eaf5ffe4acd494b24027112b7452a3a4b75aebaaab9048c874ead57b19049"}
Nov 04 00:30:03.463961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.459325    5074 generic.go:356] "Generic (PLEG): container finished" podID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerID="340aa3d3807382a66b3bc6009d52466d506b1a9ed23055aedbe2e1cacea0db25" exitCode=60
Nov 04 00:30:03.463961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.459419    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerDied","Data":"340aa3d3807382a66b3bc6009d52466d506b1a9ed23055aedbe2e1cacea0db25"}
Nov 04 00:30:03.463961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.459823    5074 scope.go:122] "RemoveContainer" containerID="340aa3d3807382a66b3bc6009d52466d506b1a9ed23055aedbe2e1cacea0db25"
Nov 04 00:30:03.540914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.540856    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" podStartSLOduration=3.540835789 podStartE2EDuration="3.540835789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.466929649 +0000 UTC m=+3668.566529794" watchObservedRunningTime="2026-11-04 00:30:03.540835789 +0000 UTC m=+3668.640435933"
Nov 04 00:30:03.541285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.541241    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" podStartSLOduration=3.541233148 podStartE2EDuration="3.541233148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.504527244 +0000 UTC m=+3668.604127389" watchObservedRunningTime="2026-11-04 00:30:03.541233148 +0000 UTC m=+3668.640833292"
Nov 04 00:30:03.563693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.563641    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" podStartSLOduration=3.563624672 podStartE2EDuration="3.563624672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.560888933 +0000 UTC m=+3668.660489073" watchObservedRunningTime="2026-11-04 00:30:03.563624672 +0000 UTC m=+3668.663224822"
Nov 04 00:30:03.585809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.585760    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" podStartSLOduration=3.585732397 podStartE2EDuration="3.585732397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.581430113 +0000 UTC m=+3668.681030260" watchObservedRunningTime="2026-11-04 00:30:03.585732397 +0000 UTC m=+3668.685332542"
Nov 04 00:30:03.610974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.610915    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" podStartSLOduration=3.610900015 podStartE2EDuration="3.610900015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.607237999 +0000 UTC m=+3668.706838144" watchObservedRunningTime="2026-11-04 00:30:03.610900015 +0000 UTC m=+3668.710500159"
Nov 04 00:30:03.629877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.629815    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" podStartSLOduration=3.6297999130000003 podStartE2EDuration="3.629799913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.629628279 +0000 UTC m=+3668.729228427" watchObservedRunningTime="2026-11-04 00:30:03.629799913 +0000 UTC m=+3668.729400058"
Nov 04 00:30:03.651008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.650950    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" podStartSLOduration=3.6509367839999998 podStartE2EDuration="3.650936784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.64871029 +0000 UTC m=+3668.748310436" watchObservedRunningTime="2026-11-04 00:30:03.650936784 +0000 UTC m=+3668.750536964"
Nov 04 00:30:03.675048 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.674993    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" podStartSLOduration=3.674978832 podStartE2EDuration="3.674978832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.67314473 +0000 UTC m=+3668.772744875" watchObservedRunningTime="2026-11-04 00:30:03.674978832 +0000 UTC m=+3668.774578976"
Nov 04 00:30:03.687032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.687000    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:03.725478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.725415    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" podStartSLOduration=3.72539521 podStartE2EDuration="3.72539521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.703952264 +0000 UTC m=+3668.803552410" watchObservedRunningTime="2026-11-04 00:30:03.72539521 +0000 UTC m=+3668.824995355"
Nov 04 00:30:03.781089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.781035    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" podStartSLOduration=3.781019925 podStartE2EDuration="3.781019925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.760595371 +0000 UTC m=+3668.860195516" watchObservedRunningTime="2026-11-04 00:30:03.781019925 +0000 UTC m=+3668.880620132"
Nov 04 00:30:03.781400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:03.781369    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" podStartSLOduration=3.781359767 podStartE2EDuration="3.781359767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.779262535 +0000 UTC m=+3668.878862676" watchObservedRunningTime="2026-11-04 00:30:03.781359767 +0000 UTC m=+3668.880959911"
Nov 04 00:30:04.048684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.048641    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"]
Nov 04 00:30:04.093932 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:04.093851    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bb3b247_d681_4d72_827a_0194a17f45af.slice/crio-46c0a039dc1ca3ecdf375c3b842d47f9f93a4241462b2cb8a826a1ab31ab72ea WatchSource:0}: Error finding container 46c0a039dc1ca3ecdf375c3b842d47f9f93a4241462b2cb8a826a1ab31ab72ea: Status 404 returned error can't find the container with id 46c0a039dc1ca3ecdf375c3b842d47f9f93a4241462b2cb8a826a1ab31ab72ea
Nov 04 00:30:04.472352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.472314    5074 generic.go:356] "Generic (PLEG): container finished" podID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerID="1557e62413e2bbb40aeee3f20f3436bba8d8a6fe1204a543cebf7b63e2210c47" exitCode=60
Nov 04 00:30:04.472850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.472397    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerDied","Data":"1557e62413e2bbb40aeee3f20f3436bba8d8a6fe1204a543cebf7b63e2210c47"}
Nov 04 00:30:04.472850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.472837    5074 scope.go:122] "RemoveContainer" containerID="1557e62413e2bbb40aeee3f20f3436bba8d8a6fe1204a543cebf7b63e2210c47"
Nov 04 00:30:04.474355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.474334    5074 generic.go:356] "Generic (PLEG): container finished" podID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerID="8d4755236d9afe0e0a292f34bf71c91539fd78d5ad88c1401240a210496d34e5" exitCode=60
Nov 04 00:30:04.474466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.474392    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerDied","Data":"8d4755236d9afe0e0a292f34bf71c91539fd78d5ad88c1401240a210496d34e5"}
Nov 04 00:30:04.474818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.474797    5074 scope.go:122] "RemoveContainer" containerID="8d4755236d9afe0e0a292f34bf71c91539fd78d5ad88c1401240a210496d34e5"
Nov 04 00:30:04.476803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.476779    5074 generic.go:356] "Generic (PLEG): container finished" podID="4c870d06-ff21-4c4e-89fc-884641120889" containerID="b5a298e448b0e1dc55bb2e017dd48c78bd92ca0100422e6749b1b8b2cf06cb4d" exitCode=60
Nov 04 00:30:04.476896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.476853    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerDied","Data":"b5a298e448b0e1dc55bb2e017dd48c78bd92ca0100422e6749b1b8b2cf06cb4d"}
Nov 04 00:30:04.477143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.477129    5074 scope.go:122] "RemoveContainer" containerID="b5a298e448b0e1dc55bb2e017dd48c78bd92ca0100422e6749b1b8b2cf06cb4d"
Nov 04 00:30:04.479323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.479306    5074 generic.go:356] "Generic (PLEG): container finished" podID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerID="4d91ce44652937d7176b397bbc2e25d6ca22bd04d0797a04df1d268aa5f79f05" exitCode=60
Nov 04 00:30:04.479409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.479389    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" event={"ID":"24b7122f-745a-4828-b1c1-17a3ac6baa04","Type":"ContainerDied","Data":"4d91ce44652937d7176b397bbc2e25d6ca22bd04d0797a04df1d268aa5f79f05"}
Nov 04 00:30:04.479595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.479574    5074 scope.go:122] "RemoveContainer" containerID="4d91ce44652937d7176b397bbc2e25d6ca22bd04d0797a04df1d268aa5f79f05"
Nov 04 00:30:04.481850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.481396    5074 generic.go:356] "Generic (PLEG): container finished" podID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerID="4e4ef006ed6738351ee3c13feb35e247dccd8cb9b8b4197f077d145ed20462a5" exitCode=60
Nov 04 00:30:04.481850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.481498    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerDied","Data":"4e4ef006ed6738351ee3c13feb35e247dccd8cb9b8b4197f077d145ed20462a5"}
Nov 04 00:30:04.481850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.481818    5074 scope.go:122] "RemoveContainer" containerID="4e4ef006ed6738351ee3c13feb35e247dccd8cb9b8b4197f077d145ed20462a5"
Nov 04 00:30:04.483166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.483142    5074 generic.go:356] "Generic (PLEG): container finished" podID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerID="6ac09aaa45c9c06230eb8d21905822eadf84ea1fde792f03bcdb4453f1c33769" exitCode=60
Nov 04 00:30:04.483269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.483209    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerDied","Data":"6ac09aaa45c9c06230eb8d21905822eadf84ea1fde792f03bcdb4453f1c33769"}
Nov 04 00:30:04.483429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.483415    5074 scope.go:122] "RemoveContainer" containerID="6ac09aaa45c9c06230eb8d21905822eadf84ea1fde792f03bcdb4453f1c33769"
Nov 04 00:30:04.485037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.485018    5074 generic.go:356] "Generic (PLEG): container finished" podID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerID="7a51956e1c212f468083318310fad88d0772f6468dbcea717d0fbbb839923f71" exitCode=60
Nov 04 00:30:04.485159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.485076    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerDied","Data":"7a51956e1c212f468083318310fad88d0772f6468dbcea717d0fbbb839923f71"}
Nov 04 00:30:04.485355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.485339    5074 scope.go:122] "RemoveContainer" containerID="7a51956e1c212f468083318310fad88d0772f6468dbcea717d0fbbb839923f71"
Nov 04 00:30:04.488258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.488236    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr" event={"ID":"7bb3b247-d681-4d72-827a-0194a17f45af","Type":"ContainerStarted","Data":"c38f7ec7ffb3def2f5dab70e1c9e76b5df520febf05dfa3f7710e38493e046af"}
Nov 04 00:30:04.488388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.488270    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr" event={"ID":"7bb3b247-d681-4d72-827a-0194a17f45af","Type":"ContainerStarted","Data":"46c0a039dc1ca3ecdf375c3b842d47f9f93a4241462b2cb8a826a1ab31ab72ea"}
Nov 04 00:30:04.501643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.501607    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"]
Nov 04 00:30:04.509847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.509818    5074 generic.go:356] "Generic (PLEG): container finished" podID="01411085-744b-4c3a-b58b-d995b0497099" containerID="88ccd7a777ad6d6a58ba795d5fdf003a35173e476572ad8f545f0ad8d86fdb06" exitCode=60
Nov 04 00:30:04.509945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.509930    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerDied","Data":"88ccd7a777ad6d6a58ba795d5fdf003a35173e476572ad8f545f0ad8d86fdb06"}
Nov 04 00:30:04.510181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.510159    5074 scope.go:122] "RemoveContainer" containerID="88ccd7a777ad6d6a58ba795d5fdf003a35173e476572ad8f545f0ad8d86fdb06"
Nov 04 00:30:04.512640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.512617    5074 generic.go:356] "Generic (PLEG): container finished" podID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerID="c65736aa7e0525135cd0170f7dfed42a374504a2234fa36475f90b8d26b34b91" exitCode=60
Nov 04 00:30:04.513199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.513022    5074 scope.go:122] "RemoveContainer" containerID="c65736aa7e0525135cd0170f7dfed42a374504a2234fa36475f90b8d26b34b91"
Nov 04 00:30:04.513386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.513361    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerDied","Data":"c65736aa7e0525135cd0170f7dfed42a374504a2234fa36475f90b8d26b34b91"}
Nov 04 00:30:04.515188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.515164    5074 generic.go:356] "Generic (PLEG): container finished" podID="361534ef-f26c-481e-80da-b758e56df5c0" containerID="97e92372680c0d288506fa591bed4ee1d6ce7c0129e4f46b120fe1c8b22b0bbe" exitCode=60
Nov 04 00:30:04.515308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.515268    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerDied","Data":"97e92372680c0d288506fa591bed4ee1d6ce7c0129e4f46b120fe1c8b22b0bbe"}
Nov 04 00:30:04.515469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.515447    5074 scope.go:122] "RemoveContainer" containerID="97e92372680c0d288506fa591bed4ee1d6ce7c0129e4f46b120fe1c8b22b0bbe"
Nov 04 00:30:04.531002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.530980    5074 generic.go:356] "Generic (PLEG): container finished" podID="484c335d-2c01-4c31-a93a-528c2361fb96" containerID="9468956e4083707069cf5d454cc6ed9eae89307ab2ddc5b84adc9e1d8e337491" exitCode=60
Nov 04 00:30:04.531177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.531159    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerDied","Data":"9468956e4083707069cf5d454cc6ed9eae89307ab2ddc5b84adc9e1d8e337491"}
Nov 04 00:30:04.531594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.531576    5074 scope.go:122] "RemoveContainer" containerID="9468956e4083707069cf5d454cc6ed9eae89307ab2ddc5b84adc9e1d8e337491"
Nov 04 00:30:04.534774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.534430    5074 generic.go:356] "Generic (PLEG): container finished" podID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerID="4c310a680ebd4d614d37efe26ff00d2117aa2ff8803dbf1a9717be9bb770f08d" exitCode=60
Nov 04 00:30:04.534774 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.534626    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" event={"ID":"590353e1-672c-467b-8a8d-ecc4cf08bb55","Type":"ContainerDied","Data":"4c310a680ebd4d614d37efe26ff00d2117aa2ff8803dbf1a9717be9bb770f08d"}
Nov 04 00:30:04.535438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.535153    5074 scope.go:122] "RemoveContainer" containerID="4c310a680ebd4d614d37efe26ff00d2117aa2ff8803dbf1a9717be9bb770f08d"
Nov 04 00:30:04.537095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.537076    5074 generic.go:356] "Generic (PLEG): container finished" podID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerID="efbea6a6ce827cbb0752bae6300bc93a4ff3e0599bc243ff9ab79222d864e577" exitCode=60
Nov 04 00:30:04.537419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.537204    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerDied","Data":"efbea6a6ce827cbb0752bae6300bc93a4ff3e0599bc243ff9ab79222d864e577"}
Nov 04 00:30:04.537543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.537518    5074 scope.go:122] "RemoveContainer" containerID="efbea6a6ce827cbb0752bae6300bc93a4ff3e0599bc243ff9ab79222d864e577"
Nov 04 00:30:04.539938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.539918    5074 generic.go:356] "Generic (PLEG): container finished" podID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerID="58294aca6587648da14700e37967df8657b425deccd5a638ad6b4bd45071b8a8" exitCode=60
Nov 04 00:30:04.540382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.540056    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" event={"ID":"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e","Type":"ContainerDied","Data":"58294aca6587648da14700e37967df8657b425deccd5a638ad6b4bd45071b8a8"}
Nov 04 00:30:04.540531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.540308    5074 scope.go:122] "RemoveContainer" containerID="58294aca6587648da14700e37967df8657b425deccd5a638ad6b4bd45071b8a8"
Nov 04 00:30:04.544334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.544319    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerID="9661ffe763edc6dd1d2720da56402f3765269945263f998717d6c969085d888b" exitCode=60
Nov 04 00:30:04.544797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.544449    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerDied","Data":"9661ffe763edc6dd1d2720da56402f3765269945263f998717d6c969085d888b"}
Nov 04 00:30:04.544881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.544621    5074 scope.go:122] "RemoveContainer" containerID="9661ffe763edc6dd1d2720da56402f3765269945263f998717d6c969085d888b"
Nov 04 00:30:04.547021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.547006    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerID="4f0eaf5ffe4acd494b24027112b7452a3a4b75aebaaab9048c874ead57b19049" exitCode=60
Nov 04 00:30:04.547298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.547143    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerDied","Data":"4f0eaf5ffe4acd494b24027112b7452a3a4b75aebaaab9048c874ead57b19049"}
Nov 04 00:30:04.547547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.547534    5074 scope.go:122] "RemoveContainer" containerID="4f0eaf5ffe4acd494b24027112b7452a3a4b75aebaaab9048c874ead57b19049"
Nov 04 00:30:04.687588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.687560    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:04.940604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:04.940519    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr" podStartSLOduration=1.940500515 podStartE2EDuration="1.940500515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:04.928582399 +0000 UTC m=+3670.028182555" watchObservedRunningTime="2026-11-04 00:30:04.940500515 +0000 UTC m=+3670.040100661"
Nov 04 00:30:05.559188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:05.559096    5074 generic.go:356] "Generic (PLEG): container finished" podID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerID="b4e371cf2a7030140817dc9378df3e6a29d6321c5031583e933e32c2dfbaf0eb" exitCode=60
Nov 04 00:30:05.559634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:05.559211    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerDied","Data":"b4e371cf2a7030140817dc9378df3e6a29d6321c5031583e933e32c2dfbaf0eb"}
Nov 04 00:30:05.559634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:05.559255    5074 scope.go:122] "RemoveContainer" containerID="340aa3d3807382a66b3bc6009d52466d506b1a9ed23055aedbe2e1cacea0db25"
Nov 04 00:30:05.559634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:05.559626    5074 scope.go:122] "RemoveContainer" containerID="b4e371cf2a7030140817dc9378df3e6a29d6321c5031583e933e32c2dfbaf0eb"
Nov 04 00:30:05.559953 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:05.559931    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd_odh-ai-gateway-infra(a2284a42-2ab9-4469-b6c4-60a7e18b6ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2"
Nov 04 00:30:05.563171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:05.563140    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerStarted","Data":"9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb"}
Nov 04 00:30:05.565515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:05.565474    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerStarted","Data":"ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca"}
Nov 04 00:30:06.576721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.576694    5074 scope.go:122] "RemoveContainer" containerID="b4e371cf2a7030140817dc9378df3e6a29d6321c5031583e933e32c2dfbaf0eb"
Nov 04 00:30:06.577162 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.577059    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd_odh-ai-gateway-infra(a2284a42-2ab9-4469-b6c4-60a7e18b6ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2"
Nov 04 00:30:06.578520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.578493    5074 generic.go:356] "Generic (PLEG): container finished" podID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerID="9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb" exitCode=60
Nov 04 00:30:06.578638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.578545    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerDied","Data":"9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb"}
Nov 04 00:30:06.578638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.578605    5074 scope.go:122] "RemoveContainer" containerID="1557e62413e2bbb40aeee3f20f3436bba8d8a6fe1204a543cebf7b63e2210c47"
Nov 04 00:30:06.578844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.578825    5074 scope.go:122] "RemoveContainer" containerID="9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb"
Nov 04 00:30:06.579120 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.579083    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk_odh-ai-gateway-infra(c92c8522-e3a0-4745-a92e-299fc4e30a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a"
Nov 04 00:30:06.580597 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.580574    5074 generic.go:356] "Generic (PLEG): container finished" podID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerID="05cee743f23344ebda53791091a1c20e12e8d1aaecdb1bfe8d1725537baf99b4" exitCode=60
Nov 04 00:30:06.580707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.580657    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerDied","Data":"05cee743f23344ebda53791091a1c20e12e8d1aaecdb1bfe8d1725537baf99b4"}
Nov 04 00:30:06.580926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.580904    5074 scope.go:122] "RemoveContainer" containerID="05cee743f23344ebda53791091a1c20e12e8d1aaecdb1bfe8d1725537baf99b4"
Nov 04 00:30:06.581190 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.581165    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq_odh-ai-gateway-infra(ac6e6048-8127-41d1-b521-4661c5462b8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f"
Nov 04 00:30:06.582832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.582805    5074 generic.go:356] "Generic (PLEG): container finished" podID="4c870d06-ff21-4c4e-89fc-884641120889" containerID="2de676db08205f41a3c92d622b202e7b418914f74263ccf4d00c519e6359c832" exitCode=60
Nov 04 00:30:06.582913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.582875    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerDied","Data":"2de676db08205f41a3c92d622b202e7b418914f74263ccf4d00c519e6359c832"}
Nov 04 00:30:06.583165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.583142    5074 scope.go:122] "RemoveContainer" containerID="2de676db08205f41a3c92d622b202e7b418914f74263ccf4d00c519e6359c832"
Nov 04 00:30:06.583443 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.583412    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm_odh-ai-gateway-infra(4c870d06-ff21-4c4e-89fc-884641120889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" podUID="4c870d06-ff21-4c4e-89fc-884641120889"
Nov 04 00:30:06.584833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.584812    5074 generic.go:356] "Generic (PLEG): container finished" podID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerID="7046c53b356b51b5cf43519d309eedfa79c437128043c3e1b03ff00e5ca0e296" exitCode=60
Nov 04 00:30:06.584940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.584835    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" event={"ID":"24b7122f-745a-4828-b1c1-17a3ac6baa04","Type":"ContainerDied","Data":"7046c53b356b51b5cf43519d309eedfa79c437128043c3e1b03ff00e5ca0e296"}
Nov 04 00:30:06.585182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.585162    5074 scope.go:122] "RemoveContainer" containerID="7046c53b356b51b5cf43519d309eedfa79c437128043c3e1b03ff00e5ca0e296"
Nov 04 00:30:06.585544 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.585516    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr_odh-ai-gateway-infra(24b7122f-745a-4828-b1c1-17a3ac6baa04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04"
Nov 04 00:30:06.587584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.587557    5074 generic.go:356] "Generic (PLEG): container finished" podID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerID="fab26d1950a3262560ea316b9b53d75c1462f2dda6d828a22f6a89da404255e5" exitCode=60
Nov 04 00:30:06.587710 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.587694    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerDied","Data":"fab26d1950a3262560ea316b9b53d75c1462f2dda6d828a22f6a89da404255e5"}
Nov 04 00:30:06.588034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.588015    5074 scope.go:122] "RemoveContainer" containerID="fab26d1950a3262560ea316b9b53d75c1462f2dda6d828a22f6a89da404255e5"
Nov 04 00:30:06.588260 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.588238    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g_odh-ai-gateway-infra(e62769d7-dfe2-4b00-9065-b1a4294bbc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13"
Nov 04 00:30:06.589650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.589628    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerStarted","Data":"7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a"}
Nov 04 00:30:06.591493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.591475    5074 generic.go:356] "Generic (PLEG): container finished" podID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerID="90208ee691c0f8e0cb82433f33e4aba3c2c3c51fb7b76545af6b8796eb1cf70d" exitCode=60
Nov 04 00:30:06.591587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.591549    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerDied","Data":"90208ee691c0f8e0cb82433f33e4aba3c2c3c51fb7b76545af6b8796eb1cf70d"}
Nov 04 00:30:06.591802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.591783    5074 scope.go:122] "RemoveContainer" containerID="90208ee691c0f8e0cb82433f33e4aba3c2c3c51fb7b76545af6b8796eb1cf70d"
Nov 04 00:30:06.592022 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.591998    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c_odh-ai-gateway-infra(b3b8bc1c-80fd-4dda-b44c-6996cc16fe12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12"
Nov 04 00:30:06.593533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.593511    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerStarted","Data":"3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c"}
Nov 04 00:30:06.595322 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.595300    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerStarted","Data":"46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2"}
Nov 04 00:30:06.597080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.597059    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerStarted","Data":"9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f"}
Nov 04 00:30:06.600594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.600546    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerStarted","Data":"56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a"}
Nov 04 00:30:06.604403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.603354    5074 generic.go:356] "Generic (PLEG): container finished" podID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerID="a9d9d4e385e2b0d4df4f26b65f35a8b2ee42ddfc3df321a8bdfc1fc68b4f07c0" exitCode=60
Nov 04 00:30:06.604403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.603432    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" event={"ID":"590353e1-672c-467b-8a8d-ecc4cf08bb55","Type":"ContainerDied","Data":"a9d9d4e385e2b0d4df4f26b65f35a8b2ee42ddfc3df321a8bdfc1fc68b4f07c0"}
Nov 04 00:30:06.605256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.604897    5074 scope.go:122] "RemoveContainer" containerID="a9d9d4e385e2b0d4df4f26b65f35a8b2ee42ddfc3df321a8bdfc1fc68b4f07c0"
Nov 04 00:30:06.605256 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.605177    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f_odh-ai-gateway-infra(590353e1-672c-467b-8a8d-ecc4cf08bb55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55"
Nov 04 00:30:06.609517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.609489    5074 generic.go:356] "Generic (PLEG): container finished" podID="11a6425b-0808-4542-8c65-d16448a10baf" containerID="ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca" exitCode=60
Nov 04 00:30:06.610593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.609791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerDied","Data":"ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca"}
Nov 04 00:30:06.610593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.609905    5074 scope.go:122] "RemoveContainer" containerID="ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca"
Nov 04 00:30:06.610593 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.610169    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4_odh-ai-gateway-infra(11a6425b-0808-4542-8c65-d16448a10baf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" podUID="11a6425b-0808-4542-8c65-d16448a10baf"
Nov 04 00:30:06.613699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.613672    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerStarted","Data":"826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e"}
Nov 04 00:30:06.616224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.616197    5074 generic.go:356] "Generic (PLEG): container finished" podID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerID="5c991b58af5c2bda108241bb9a758f29e7a1f75065de5217967439b775c85791" exitCode=60
Nov 04 00:30:06.616301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.616269    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" event={"ID":"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e","Type":"ContainerDied","Data":"5c991b58af5c2bda108241bb9a758f29e7a1f75065de5217967439b775c85791"}
Nov 04 00:30:06.616613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.616595    5074 scope.go:122] "RemoveContainer" containerID="5c991b58af5c2bda108241bb9a758f29e7a1f75065de5217967439b775c85791"
Nov 04 00:30:06.616878 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.616856    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn_odh-ai-gateway-infra(5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e"
Nov 04 00:30:06.627421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.627394    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerStarted","Data":"f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"}
Nov 04 00:30:06.627795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.627773    5074 scope.go:122] "RemoveContainer" containerID="f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"
Nov 04 00:30:06.628088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.628064    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx_odh-ai-gateway-infra(e4961c11-d4e7-4ccd-99e8-13766e5fea74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74"
Nov 04 00:30:06.630890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.630850    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerStarted","Data":"d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"}
Nov 04 00:30:06.631156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:06.631138    5074 scope.go:122] "RemoveContainer" containerID="d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"
Nov 04 00:30:06.631410 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:06.631389    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx_odh-ai-gateway-infra(a3fd1187-3dac-40d1-9472-3ff3c92d6512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512"
Nov 04 00:30:07.161264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.161175    5074 scope.go:122] "RemoveContainer" containerID="8d4755236d9afe0e0a292f34bf71c91539fd78d5ad88c1401240a210496d34e5"
Nov 04 00:30:07.175879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.175855    5074 scope.go:122] "RemoveContainer" containerID="b5a298e448b0e1dc55bb2e017dd48c78bd92ca0100422e6749b1b8b2cf06cb4d"
Nov 04 00:30:07.191728 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.191708    5074 scope.go:122] "RemoveContainer" containerID="4d91ce44652937d7176b397bbc2e25d6ca22bd04d0797a04df1d268aa5f79f05"
Nov 04 00:30:07.206405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.206386    5074 scope.go:122] "RemoveContainer" containerID="4e4ef006ed6738351ee3c13feb35e247dccd8cb9b8b4197f077d145ed20462a5"
Nov 04 00:30:07.222735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.222715    5074 scope.go:122] "RemoveContainer" containerID="7a51956e1c212f468083318310fad88d0772f6468dbcea717d0fbbb839923f71"
Nov 04 00:30:07.237102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.237083    5074 scope.go:122] "RemoveContainer" containerID="4c310a680ebd4d614d37efe26ff00d2117aa2ff8803dbf1a9717be9bb770f08d"
Nov 04 00:30:07.251714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.251689    5074 scope.go:122] "RemoveContainer" containerID="9858583eb8ca37e8a4b1ecee0984add60dd4231ddf68e48677971f38a18b71ff"
Nov 04 00:30:07.268467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.268446    5074 scope.go:122] "RemoveContainer" containerID="58294aca6587648da14700e37967df8657b425deccd5a638ad6b4bd45071b8a8"
Nov 04 00:30:07.642540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.642356    5074 scope.go:122] "RemoveContainer" containerID="9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb"
Nov 04 00:30:07.643547 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.642821    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk_odh-ai-gateway-infra(c92c8522-e3a0-4745-a92e-299fc4e30a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a"
Nov 04 00:30:07.644363 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.644335    5074 scope.go:122] "RemoveContainer" containerID="05cee743f23344ebda53791091a1c20e12e8d1aaecdb1bfe8d1725537baf99b4"
Nov 04 00:30:07.644712 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.644679    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq_odh-ai-gateway-infra(ac6e6048-8127-41d1-b521-4661c5462b8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f"
Nov 04 00:30:07.646173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.646151    5074 scope.go:122] "RemoveContainer" containerID="2de676db08205f41a3c92d622b202e7b418914f74263ccf4d00c519e6359c832"
Nov 04 00:30:07.646519 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.646488    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm_odh-ai-gateway-infra(4c870d06-ff21-4c4e-89fc-884641120889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" podUID="4c870d06-ff21-4c4e-89fc-884641120889"
Nov 04 00:30:07.647988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.647969    5074 scope.go:122] "RemoveContainer" containerID="7046c53b356b51b5cf43519d309eedfa79c437128043c3e1b03ff00e5ca0e296"
Nov 04 00:30:07.648222 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.648198    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr_odh-ai-gateway-infra(24b7122f-745a-4828-b1c1-17a3ac6baa04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04"
Nov 04 00:30:07.655021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.654996    5074 scope.go:122] "RemoveContainer" containerID="fab26d1950a3262560ea316b9b53d75c1462f2dda6d828a22f6a89da404255e5"
Nov 04 00:30:07.655304 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.655280    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g_odh-ai-gateway-infra(e62769d7-dfe2-4b00-9065-b1a4294bbc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13"
Nov 04 00:30:07.656575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.656554    5074 generic.go:356] "Generic (PLEG): container finished" podID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerID="7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a" exitCode=60
Nov 04 00:30:07.656667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.656620    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerDied","Data":"7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a"}
Nov 04 00:30:07.656761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.656670    5074 scope.go:122] "RemoveContainer" containerID="6ac09aaa45c9c06230eb8d21905822eadf84ea1fde792f03bcdb4453f1c33769"
Nov 04 00:30:07.656946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.656926    5074 scope.go:122] "RemoveContainer" containerID="7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a"
Nov 04 00:30:07.657235 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.657208    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk_odh-ai-gateway-infra(f9ba160e-304a-46cd-aad9-4b0151a85b44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44"
Nov 04 00:30:07.659335 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.659317    5074 scope.go:122] "RemoveContainer" containerID="90208ee691c0f8e0cb82433f33e4aba3c2c3c51fb7b76545af6b8796eb1cf70d"
Nov 04 00:30:07.659542 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.659521    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c_odh-ai-gateway-infra(b3b8bc1c-80fd-4dda-b44c-6996cc16fe12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12"
Nov 04 00:30:07.661453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.661429    5074 generic.go:356] "Generic (PLEG): container finished" podID="01411085-744b-4c3a-b58b-d995b0497099" containerID="3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c" exitCode=60
Nov 04 00:30:07.661544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.661454    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerDied","Data":"3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c"}
Nov 04 00:30:07.661795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.661720    5074 scope.go:122] "RemoveContainer" containerID="3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c"
Nov 04 00:30:07.662000 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.661979    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw_odh-ai-gateway-infra(01411085-744b-4c3a-b58b-d995b0497099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" podUID="01411085-744b-4c3a-b58b-d995b0497099"
Nov 04 00:30:07.664419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.664391    5074 generic.go:356] "Generic (PLEG): container finished" podID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerID="46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2" exitCode=60
Nov 04 00:30:07.664505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.664490    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerDied","Data":"46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2"}
Nov 04 00:30:07.664873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.664853    5074 scope.go:122] "RemoveContainer" containerID="46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2"
Nov 04 00:30:07.665152 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.665129    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9_odh-ai-gateway-infra(b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3"
Nov 04 00:30:07.667938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.667886    5074 generic.go:356] "Generic (PLEG): container finished" podID="361534ef-f26c-481e-80da-b758e56df5c0" containerID="9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f" exitCode=60
Nov 04 00:30:07.667938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.667920    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerDied","Data":"9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f"}
Nov 04 00:30:07.668281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.668217    5074 scope.go:122] "RemoveContainer" containerID="9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f"
Nov 04 00:30:07.668495 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.668470    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb_odh-ai-gateway-infra(361534ef-f26c-481e-80da-b758e56df5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" podUID="361534ef-f26c-481e-80da-b758e56df5c0"
Nov 04 00:30:07.671029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.671012    5074 generic.go:356] "Generic (PLEG): container finished" podID="484c335d-2c01-4c31-a93a-528c2361fb96" containerID="56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a" exitCode=60
Nov 04 00:30:07.671131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.671072    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerDied","Data":"56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a"}
Nov 04 00:30:07.671309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.671291    5074 scope.go:122] "RemoveContainer" containerID="56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a"
Nov 04 00:30:07.671535 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.671508    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl_odh-ai-gateway-infra(484c335d-2c01-4c31-a93a-528c2361fb96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" podUID="484c335d-2c01-4c31-a93a-528c2361fb96"
Nov 04 00:30:07.672932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.672914    5074 scope.go:122] "RemoveContainer" containerID="a9d9d4e385e2b0d4df4f26b65f35a8b2ee42ddfc3df321a8bdfc1fc68b4f07c0"
Nov 04 00:30:07.673144 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.673118    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f_odh-ai-gateway-infra(590353e1-672c-467b-8a8d-ecc4cf08bb55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55"
Nov 04 00:30:07.674790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.674767    5074 scope.go:122] "RemoveContainer" containerID="ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca"
Nov 04 00:30:07.675042 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.675022    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4_odh-ai-gateway-infra(11a6425b-0808-4542-8c65-d16448a10baf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" podUID="11a6425b-0808-4542-8c65-d16448a10baf"
Nov 04 00:30:07.675992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.675976    5074 scope.go:122] "RemoveContainer" containerID="88ccd7a777ad6d6a58ba795d5fdf003a35173e476572ad8f545f0ad8d86fdb06"
Nov 04 00:30:07.677502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.677474    5074 generic.go:356] "Generic (PLEG): container finished" podID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerID="826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e" exitCode=60
Nov 04 00:30:07.677604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.677554    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerDied","Data":"826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e"}
Nov 04 00:30:07.677829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.677811    5074 scope.go:122] "RemoveContainer" containerID="826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e"
Nov 04 00:30:07.678049 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.678026    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5_odh-ai-gateway-infra(6d407d9a-345c-495f-9ab2-2f2eb28bf209)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209"
Nov 04 00:30:07.679590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.679574    5074 scope.go:122] "RemoveContainer" containerID="5c991b58af5c2bda108241bb9a758f29e7a1f75065de5217967439b775c85791"
Nov 04 00:30:07.679861 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.679833    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn_odh-ai-gateway-infra(5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e"
Nov 04 00:30:07.681604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.681582    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerID="f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb" exitCode=60
Nov 04 00:30:07.681693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.681649    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerDied","Data":"f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"}
Nov 04 00:30:07.681908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.681893    5074 scope.go:122] "RemoveContainer" containerID="f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"
Nov 04 00:30:07.682153 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.682114    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx_odh-ai-gateway-infra(e4961c11-d4e7-4ccd-99e8-13766e5fea74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74"
Nov 04 00:30:07.684000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.683973    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerID="d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946" exitCode=60
Nov 04 00:30:07.684117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.684049    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerDied","Data":"d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"}
Nov 04 00:30:07.684261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.684247    5074 scope.go:122] "RemoveContainer" containerID="d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"
Nov 04 00:30:07.684479 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:07.684458    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx_odh-ai-gateway-infra(a3fd1187-3dac-40d1-9472-3ff3c92d6512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512"
Nov 04 00:30:07.694835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.694816    5074 scope.go:122] "RemoveContainer" containerID="c65736aa7e0525135cd0170f7dfed42a374504a2234fa36475f90b8d26b34b91"
Nov 04 00:30:07.711421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.711392    5074 scope.go:122] "RemoveContainer" containerID="97e92372680c0d288506fa591bed4ee1d6ce7c0129e4f46b120fe1c8b22b0bbe"
Nov 04 00:30:07.728057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.728033    5074 scope.go:122] "RemoveContainer" containerID="9468956e4083707069cf5d454cc6ed9eae89307ab2ddc5b84adc9e1d8e337491"
Nov 04 00:30:07.745014 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.744841    5074 scope.go:122] "RemoveContainer" containerID="efbea6a6ce827cbb0752bae6300bc93a4ff3e0599bc243ff9ab79222d864e577"
Nov 04 00:30:07.761392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.761362    5074 scope.go:122] "RemoveContainer" containerID="9661ffe763edc6dd1d2720da56402f3765269945263f998717d6c969085d888b"
Nov 04 00:30:07.809025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:07.809002    5074 scope.go:122] "RemoveContainer" containerID="4f0eaf5ffe4acd494b24027112b7452a3a4b75aebaaab9048c874ead57b19049"
Nov 04 00:30:08.696417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.696388    5074 scope.go:122] "RemoveContainer" containerID="7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a"
Nov 04 00:30:08.696929 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.696754    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk_odh-ai-gateway-infra(f9ba160e-304a-46cd-aad9-4b0151a85b44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44"
Nov 04 00:30:08.698762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.698430    5074 scope.go:122] "RemoveContainer" containerID="3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c"
Nov 04 00:30:08.698872 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.698786    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw_odh-ai-gateway-infra(01411085-744b-4c3a-b58b-d995b0497099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" podUID="01411085-744b-4c3a-b58b-d995b0497099"
Nov 04 00:30:08.700380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.700358    5074 scope.go:122] "RemoveContainer" containerID="46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2"
Nov 04 00:30:08.700763 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.700701    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9_odh-ai-gateway-infra(b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3"
Nov 04 00:30:08.702692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.702668    5074 scope.go:122] "RemoveContainer" containerID="9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f"
Nov 04 00:30:08.702978 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.702950    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb_odh-ai-gateway-infra(361534ef-f26c-481e-80da-b758e56df5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" podUID="361534ef-f26c-481e-80da-b758e56df5c0"
Nov 04 00:30:08.707268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.705300    5074 scope.go:122] "RemoveContainer" containerID="56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a"
Nov 04 00:30:08.707268 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.705579    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl_odh-ai-gateway-infra(484c335d-2c01-4c31-a93a-528c2361fb96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" podUID="484c335d-2c01-4c31-a93a-528c2361fb96"
Nov 04 00:30:08.722180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.722155    5074 scope.go:122] "RemoveContainer" containerID="826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e"
Nov 04 00:30:08.722984 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.722955    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5_odh-ai-gateway-infra(6d407d9a-345c-495f-9ab2-2f2eb28bf209)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209"
Nov 04 00:30:08.735244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.735214    5074 scope.go:122] "RemoveContainer" containerID="f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"
Nov 04 00:30:08.735757 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.735650    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx_odh-ai-gateway-infra(e4961c11-d4e7-4ccd-99e8-13766e5fea74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74"
Nov 04 00:30:08.741228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:08.741208    5074 scope.go:122] "RemoveContainer" containerID="d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"
Nov 04 00:30:08.741798 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:08.741769    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx_odh-ai-gateway-infra(a3fd1187-3dac-40d1-9472-3ff3c92d6512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512"
Nov 04 00:30:09.502981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.502932    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj" podUID="d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" containerName="payload-processing" containerID="cri-o://46d72d277097818c4904681e7d34390eff614ac5415688a04e55fafe34c60941" gracePeriod=25
Nov 04 00:30:09.751487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.751447    5074 generic.go:356] "Generic (PLEG): container finished" podID="d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" containerID="46d72d277097818c4904681e7d34390eff614ac5415688a04e55fafe34c60941" exitCode=0
Nov 04 00:30:09.751920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.751527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj" event={"ID":"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3","Type":"ContainerDied","Data":"46d72d277097818c4904681e7d34390eff614ac5415688a04e55fafe34c60941"}
Nov 04 00:30:09.827226 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.827194    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:30:09.932528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.932486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume\") pod \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") "
Nov 04 00:30:09.932701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.932540    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6\" (UniqueName: \"kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6\") pod \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\" (UID: \"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3\") "
Nov 04 00:30:09.933011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.932987    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume" pod "d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" (UID: "d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:09.933310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.933288    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:09.934754 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:09.934709    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6" pod "d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" (UID: "d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3"). InnerVolumeSpecName "kube-api-access-4zpb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:10.035101 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:10.035055    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpb6\" (UniqueName: \"kubernetes.io/projected/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3-kube-api-access-4zpb6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:10.765993 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:10.765961    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"
Nov 04 00:30:10.766515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:10.765964    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj" event={"ID":"d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3","Type":"ContainerDied","Data":"030c0cea0d7fdd64b158df9ef603a014f93cd434e9d284c7b1951319d08e5e87"}
Nov 04 00:30:10.766515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:10.766111    5074 scope.go:122] "RemoveContainer" containerID="46d72d277097818c4904681e7d34390eff614ac5415688a04e55fafe34c60941"
Nov 04 00:30:10.795121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:10.795092    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"]
Nov 04 00:30:10.800367 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:10.800339    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9c0227c-77d7949d8b-xvdhj"]
Nov 04 00:30:11.032864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:11.032770    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" path="/var/lib/kubelet/pods/d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3/volumes"
Nov 04 00:30:13.687641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:13.687610    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:13.703823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:13.703794    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:13.809362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:13.809327    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:30:17.010841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:17.010810    5074 scope.go:122] "RemoveContainer" containerID="b4e371cf2a7030140817dc9378df3e6a29d6321c5031583e933e32c2dfbaf0eb"
Nov 04 00:30:17.011426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:17.010953    5074 scope.go:122] "RemoveContainer" containerID="46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2"
Nov 04 00:30:17.849392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:17.849354    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerStarted","Data":"48d486657bd9065ace9da328975153626fea4d8a2545507f36c38481f5400a44"}
Nov 04 00:30:17.851871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:17.851837    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerStarted","Data":"d9f74a9520a2ef8df48cee14ac4ec4ec2b86ffaa4894e2c62640674a998aaf4e"}
Nov 04 00:30:18.010318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.010287    5074 scope.go:122] "RemoveContainer" containerID="826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e"
Nov 04 00:30:18.010572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.010552    5074 scope.go:122] "RemoveContainer" containerID="7046c53b356b51b5cf43519d309eedfa79c437128043c3e1b03ff00e5ca0e296"
Nov 04 00:30:18.011112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.010879    5074 scope.go:122] "RemoveContainer" containerID="2de676db08205f41a3c92d622b202e7b418914f74263ccf4d00c519e6359c832"
Nov 04 00:30:18.011896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.011463    5074 scope.go:122] "RemoveContainer" containerID="9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb"
Nov 04 00:30:18.012344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.012275    5074 scope.go:122] "RemoveContainer" containerID="fab26d1950a3262560ea316b9b53d75c1462f2dda6d828a22f6a89da404255e5"
Nov 04 00:30:18.862456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.862415    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerStarted","Data":"7f24e2652e0a8480f4348cd9e0d832d5ce4ea69cbf451a1b4b98ab486e9d0c6b"}
Nov 04 00:30:18.864542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.864517    5074 generic.go:356] "Generic (PLEG): container finished" podID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerID="48d486657bd9065ace9da328975153626fea4d8a2545507f36c38481f5400a44" exitCode=60
Nov 04 00:30:18.864670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.864585    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerDied","Data":"48d486657bd9065ace9da328975153626fea4d8a2545507f36c38481f5400a44"}
Nov 04 00:30:18.864670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.864634    5074 scope.go:122] "RemoveContainer" containerID="b4e371cf2a7030140817dc9378df3e6a29d6321c5031583e933e32c2dfbaf0eb"
Nov 04 00:30:18.865073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.865051    5074 scope.go:122] "RemoveContainer" containerID="48d486657bd9065ace9da328975153626fea4d8a2545507f36c38481f5400a44"
Nov 04 00:30:18.865415 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:18.865391    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd_odh-ai-gateway-infra(a2284a42-2ab9-4469-b6c4-60a7e18b6ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2"
Nov 04 00:30:18.868232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.868207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerStarted","Data":"b51fe33700bd66201be6ca22950c58645504be65ddaa749bce39f7310d70594c"}
Nov 04 00:30:18.870416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.870398    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerStarted","Data":"dabfee51e63411b31008c836f372c7b947cdc649e9f6a3e7bd659dd1115e6b89"}
Nov 04 00:30:18.872674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.872651    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" event={"ID":"24b7122f-745a-4828-b1c1-17a3ac6baa04","Type":"ContainerStarted","Data":"39f609460e5fb2e06c642c94e5d4719643342dc34e737dd61d2e5c8ea2eed1a3"}
Nov 04 00:30:18.874840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.874816    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerStarted","Data":"c113febadd676042b9f12991ff94faeabfccf346ad2477cb4e58d0bc469320cd"}
Nov 04 00:30:18.876883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.876859    5074 generic.go:356] "Generic (PLEG): container finished" podID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerID="d9f74a9520a2ef8df48cee14ac4ec4ec2b86ffaa4894e2c62640674a998aaf4e" exitCode=60
Nov 04 00:30:18.876968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.876926    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerDied","Data":"d9f74a9520a2ef8df48cee14ac4ec4ec2b86ffaa4894e2c62640674a998aaf4e"}
Nov 04 00:30:18.877268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.877253    5074 scope.go:122] "RemoveContainer" containerID="d9f74a9520a2ef8df48cee14ac4ec4ec2b86ffaa4894e2c62640674a998aaf4e"
Nov 04 00:30:18.877534 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:18.877516    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9_odh-ai-gateway-infra(b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3"
Nov 04 00:30:18.881871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.881854    5074 scope.go:122] "RemoveContainer" containerID="46936caa088c7c9ad0d092c3c1f9e9808cc9a3369e77d377e72ab44ca73c0fc2"
Nov 04 00:30:18.884929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.884906    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"]
Nov 04 00:30:18.923190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.923155    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"]
Nov 04 00:30:18.953311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:18.953255    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" podStartSLOduration=18.953243071 podStartE2EDuration="18.953243071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:18.951795707 +0000 UTC m=+3684.051395850" watchObservedRunningTime="2026-11-04 00:30:18.953243071 +0000 UTC m=+3684.052843214"
Nov 04 00:30:19.028811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.021561    5074 scope.go:122] "RemoveContainer" containerID="56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a"
Nov 04 00:30:19.028811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.023877    5074 scope.go:122] "RemoveContainer" containerID="90208ee691c0f8e0cb82433f33e4aba3c2c3c51fb7b76545af6b8796eb1cf70d"
Nov 04 00:30:19.028811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.025169    5074 scope.go:122] "RemoveContainer" containerID="3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c"
Nov 04 00:30:19.028811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.025714    5074 scope.go:122] "RemoveContainer" containerID="7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a"
Nov 04 00:30:19.028811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.026874    5074 scope.go:122] "RemoveContainer" containerID="d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"
Nov 04 00:30:19.053706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.052900    5074 scope.go:122] "RemoveContainer" containerID="05cee743f23344ebda53791091a1c20e12e8d1aaecdb1bfe8d1725537baf99b4"
Nov 04 00:30:19.574070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.574032    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"]
Nov 04 00:30:19.575503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.575481    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" containerName="payload-processing"
Nov 04 00:30:19.575577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.575509    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" containerName="payload-processing"
Nov 04 00:30:19.575811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.575796    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71801ec-cc14-4c34-8d9b-6c13f5a2e0d3" containerName="payload-processing"
Nov 04 00:30:19.580932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.580905    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.587793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.587765    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"sub-test-model-15b02a-kserve-self-signed-certs\""
Nov 04 00:30:19.587943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.587811    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"default-dockercfg-n7zls\""
Nov 04 00:30:19.587943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.587883    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"kube-root-ca.crt\""
Nov 04 00:30:19.592408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.592383    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"]
Nov 04 00:30:19.685719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.685686    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"openshift-service-ca.crt\""
Nov 04 00:30:19.686534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.686503    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvwj\" (UniqueName: \"kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.686799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.686773    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.686909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.686814    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.686983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.686934    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.687046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.687022    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.687151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.687133    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.790166    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.790695    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.790753    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.790800    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.790940    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.791080    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvwj\" (UniqueName: \"kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.791714    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.794117    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.794763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.794410    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.809764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.806789    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.824767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.818511    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.837932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.837885    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvwj\" (UniqueName: \"kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj\") pod \"sub-test-model-15b02a-kserve-7cd9b5647f-h9frv\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:19.889826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.889788    5074 generic.go:356] "Generic (PLEG): container finished" podID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerID="c113febadd676042b9f12991ff94faeabfccf346ad2477cb4e58d0bc469320cd" exitCode=60
Nov 04 00:30:19.890013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.889852    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerDied","Data":"c113febadd676042b9f12991ff94faeabfccf346ad2477cb4e58d0bc469320cd"}
Nov 04 00:30:19.890013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.889915    5074 scope.go:122] "RemoveContainer" containerID="fab26d1950a3262560ea316b9b53d75c1462f2dda6d828a22f6a89da404255e5"
Nov 04 00:30:19.890472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.890449    5074 scope.go:122] "RemoveContainer" containerID="c113febadd676042b9f12991ff94faeabfccf346ad2477cb4e58d0bc469320cd"
Nov 04 00:30:19.890995 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:19.890959    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g_odh-ai-gateway-infra(e62769d7-dfe2-4b00-9065-b1a4294bbc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13"
Nov 04 00:30:19.895289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.895268    5074 generic.go:356] "Generic (PLEG): container finished" podID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerID="7f24e2652e0a8480f4348cd9e0d832d5ce4ea69cbf451a1b4b98ab486e9d0c6b" exitCode=60
Nov 04 00:30:19.895397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.895340    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerDied","Data":"7f24e2652e0a8480f4348cd9e0d832d5ce4ea69cbf451a1b4b98ab486e9d0c6b"}
Nov 04 00:30:19.895776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.895723    5074 scope.go:122] "RemoveContainer" containerID="7f24e2652e0a8480f4348cd9e0d832d5ce4ea69cbf451a1b4b98ab486e9d0c6b"
Nov 04 00:30:19.896061 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:19.896038    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5_odh-ai-gateway-infra(6d407d9a-345c-495f-9ab2-2f2eb28bf209)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209"
Nov 04 00:30:19.900851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.899498    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"]
Nov 04 00:30:19.901032 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.901011    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerDied","Data":"b51fe33700bd66201be6ca22950c58645504be65ddaa749bce39f7310d70594c"}
Nov 04 00:30:19.901450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.901432    5074 scope.go:122] "RemoveContainer" containerID="b51fe33700bd66201be6ca22950c58645504be65ddaa749bce39f7310d70594c"
Nov 04 00:30:19.901782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.899753    5074 generic.go:356] "Generic (PLEG): container finished" podID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerID="b51fe33700bd66201be6ca22950c58645504be65ddaa749bce39f7310d70594c" exitCode=60
Nov 04 00:30:19.901980 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:19.901957    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk_odh-ai-gateway-infra(c92c8522-e3a0-4745-a92e-299fc4e30a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a"
Nov 04 00:30:19.906086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.905573    5074 generic.go:356] "Generic (PLEG): container finished" podID="4c870d06-ff21-4c4e-89fc-884641120889" containerID="dabfee51e63411b31008c836f372c7b947cdc649e9f6a3e7bd659dd1115e6b89" exitCode=60
Nov 04 00:30:19.906086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.905612    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerDied","Data":"dabfee51e63411b31008c836f372c7b947cdc649e9f6a3e7bd659dd1115e6b89"}
Nov 04 00:30:19.906086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.906051    5074 scope.go:122] "RemoveContainer" containerID="dabfee51e63411b31008c836f372c7b947cdc649e9f6a3e7bd659dd1115e6b89"
Nov 04 00:30:19.906621 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:19.906450    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm_odh-ai-gateway-infra(4c870d06-ff21-4c4e-89fc-884641120889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" podUID="4c870d06-ff21-4c4e-89fc-884641120889"
Nov 04 00:30:19.909188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.909166    5074 generic.go:356] "Generic (PLEG): container finished" podID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerID="39f609460e5fb2e06c642c94e5d4719643342dc34e737dd61d2e5c8ea2eed1a3" exitCode=60
Nov 04 00:30:19.909300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.909214    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" event={"ID":"24b7122f-745a-4828-b1c1-17a3ac6baa04","Type":"ContainerDied","Data":"39f609460e5fb2e06c642c94e5d4719643342dc34e737dd61d2e5c8ea2eed1a3"}
Nov 04 00:30:19.909511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.909492    5074 scope.go:122] "RemoveContainer" containerID="39f609460e5fb2e06c642c94e5d4719643342dc34e737dd61d2e5c8ea2eed1a3"
Nov 04 00:30:19.909729 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:19.909696    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr_odh-ai-gateway-infra(24b7122f-745a-4828-b1c1-17a3ac6baa04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04"
Nov 04 00:30:19.926997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.926552    5074 scope.go:122] "RemoveContainer" containerID="826eaf6cea4d14e113c78c3bed76392cff00b2f7c1cc4eb4f22139baa8a0cb7e"
Nov 04 00:30:19.932855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.932833    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"]
Nov 04 00:30:19.951540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.951506    5074 scope.go:122] "RemoveContainer" containerID="9364dfeaa85d44bede4e1f43968954f5bf98021ff12883cd39d3a753318622fb"
Nov 04 00:30:19.965608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.965580    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"]
Nov 04 00:30:19.967648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.967627    5074 scope.go:122] "RemoveContainer" containerID="2de676db08205f41a3c92d622b202e7b418914f74263ccf4d00c519e6359c832"
Nov 04 00:30:19.986705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.986679    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"]
Nov 04 00:30:19.996005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:19.992033    5074 scope.go:122] "RemoveContainer" containerID="7046c53b356b51b5cf43519d309eedfa79c437128043c3e1b03ff00e5ca0e296"
Nov 04 00:30:20.008403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.008236    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"]
Nov 04 00:30:20.017796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.014378    5074 scope.go:122] "RemoveContainer" containerID="5c991b58af5c2bda108241bb9a758f29e7a1f75065de5217967439b775c85791"
Nov 04 00:30:20.017796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.015222    5074 scope.go:122] "RemoveContainer" containerID="ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca"
Nov 04 00:30:20.017796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.015950    5074 scope.go:122] "RemoveContainer" containerID="a9d9d4e385e2b0d4df4f26b65f35a8b2ee42ddfc3df321a8bdfc1fc68b4f07c0"
Nov 04 00:30:20.017796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.016556    5074 scope.go:122] "RemoveContainer" containerID="9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f"
Nov 04 00:30:20.019621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.019097    5074 scope.go:122] "RemoveContainer" containerID="f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"
Nov 04 00:30:20.040380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.040168    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:20.346764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.346025    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:20.437969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.436501    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6\" (UniqueName: \"kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6\") pod \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") "
Nov 04 00:30:20.437969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.436666    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca\") pod \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\" (UID: \"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3\") "
Nov 04 00:30:20.437969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.437519    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca" pod "b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" (UID: "b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:20.539480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.539443    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:20.566787 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.566734    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"]
Nov 04 00:30:20.609394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.605942    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6" pod "b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" (UID: "b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3"). InnerVolumeSpecName "kube-api-access-66lk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:20.641099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.641066    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66lk6\" (UniqueName: \"kubernetes.io/projected/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3-kube-api-access-66lk6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:20.779048 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:20.779015    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod811cb4e3_12f4_4791_9ed1_0807d4045f5e.slice/crio-63a66b55bdefccc25c52a4ceb4d5eee77f992a69450e5a077ccda68bd6beafdf WatchSource:0}: Error finding container 63a66b55bdefccc25c52a4ceb4d5eee77f992a69450e5a077ccda68bd6beafdf: Status 404 returned error can't find the container with id 63a66b55bdefccc25c52a4ceb4d5eee77f992a69450e5a077ccda68bd6beafdf
Nov 04 00:30:20.789434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.789410    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.936621    5074 generic.go:356] "Generic (PLEG): container finished" podID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerID="a552ada02007941e220f4cc7718af9d3fb5b9df1417aea0a874934d9f3ab6679" exitCode=60
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.936706    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerDied","Data":"a552ada02007941e220f4cc7718af9d3fb5b9df1417aea0a874934d9f3ab6679"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.936804    5074 scope.go:122] "RemoveContainer" containerID="7272826789b058957db4270e0266b52caf620cc4f116b63ec3097367fca3be7a"
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.937412    5074 scope.go:122] "RemoveContainer" containerID="a552ada02007941e220f4cc7718af9d3fb5b9df1417aea0a874934d9f3ab6679"
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:20.937859    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk_odh-ai-gateway-infra(f9ba160e-304a-46cd-aad9-4b0151a85b44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44"
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.940315    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerStarted","Data":"af0923b82f291fb02d9f393a6e91e1358ae61ce10bbf1828b3f9b5e80339af4b"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.941974    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" event={"ID":"811cb4e3-12f4-4791-9ed1-0807d4045f5e","Type":"ContainerStarted","Data":"63a66b55bdefccc25c52a4ceb4d5eee77f992a69450e5a077ccda68bd6beafdf"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.944133    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerStarted","Data":"a1027daa4cd47cd1b9c25ef5f0d155e134d9b6047fc5988dce096d255c6a2382"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.944642    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x\" (UniqueName: \"kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x\") pod \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") "
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.944991    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca\") pod \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\" (UID: \"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2\") "
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.946501    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9" event={"ID":"b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3","Type":"ContainerDied","Data":"2ad117073f264381fa18cc403ce87cb054bb0022afa2bde8bf80e94edd2eb0d1"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.946603    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.947010    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca" pod "a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" (UID: "a2284a42-2ab9-4469-b6c4-60a7e18b6ab2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.948252    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.949590    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerStarted","Data":"c504effeb406f646a337a28342b30f7f494a7dc352cee04afd8fa093b84ef7af"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.956354    5074 generic.go:356] "Generic (PLEG): container finished" podID="484c335d-2c01-4c31-a93a-528c2361fb96" containerID="48808a70371ea504d3ffc0f89f6bd7789cb8452e602ff731cf4605236dd5bd38" exitCode=60
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.956433    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerDied","Data":"48808a70371ea504d3ffc0f89f6bd7789cb8452e602ff731cf4605236dd5bd38"}
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.957203    5074 scope.go:122] "RemoveContainer" containerID="48808a70371ea504d3ffc0f89f6bd7789cb8452e602ff731cf4605236dd5bd38"
Nov 04 00:30:20.959029 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:20.957470    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl_odh-ai-gateway-infra(484c335d-2c01-4c31-a93a-528c2361fb96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" podUID="484c335d-2c01-4c31-a93a-528c2361fb96"
Nov 04 00:30:20.982919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.982878    5074 generic.go:356] "Generic (PLEG): container finished" podID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerID="79f4019f8fb156ab85e001cea87e883a6df83b0d2f51eec9af184bbe245a970f" exitCode=60
Nov 04 00:30:20.983063 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.983030    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerDied","Data":"79f4019f8fb156ab85e001cea87e883a6df83b0d2f51eec9af184bbe245a970f"}
Nov 04 00:30:20.983598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.983568    5074 scope.go:122] "RemoveContainer" containerID="79f4019f8fb156ab85e001cea87e883a6df83b0d2f51eec9af184bbe245a970f"
Nov 04 00:30:20.984123 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:20.984089    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx_odh-ai-gateway-infra(a3fd1187-3dac-40d1-9472-3ff3c92d6512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512"
Nov 04 00:30:20.986553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.986520    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd" event={"ID":"a2284a42-2ab9-4469-b6c4-60a7e18b6ab2","Type":"ContainerDied","Data":"c2b3f32c07b9ce0cc18f603e51efb1a446241b1e82d6d6b62e51e519b810b6be"}
Nov 04 00:30:20.986693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.986673    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"
Nov 04 00:30:20.997667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:20.997621    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerStarted","Data":"fcc57e80e3f6f44308e321540e6a45e21d53af56b18cf93fbee7adc455587824"}
Nov 04 00:30:21.005422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.005376    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x" pod "a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" (UID: "a2284a42-2ab9-4469-b6c4-60a7e18b6ab2"). InnerVolumeSpecName "kube-api-access-5cf2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:21.009230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.009209    5074 scope.go:122] "RemoveContainer" containerID="fcc57e80e3f6f44308e321540e6a45e21d53af56b18cf93fbee7adc455587824"
Nov 04 00:30:21.013020 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:21.012519    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq_odh-ai-gateway-infra(ac6e6048-8127-41d1-b521-4661c5462b8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f"
Nov 04 00:30:21.049686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.042231    5074 scope.go:122] "RemoveContainer" containerID="d9f74a9520a2ef8df48cee14ac4ec4ec2b86ffaa4894e2c62640674a998aaf4e"
Nov 04 00:30:21.058799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.052244    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cf2x\" (UniqueName: \"kubernetes.io/projected/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2-kube-api-access-5cf2x\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.266353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.265684    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:21.269940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.266800    5074 scope.go:122] "RemoveContainer" containerID="56e36a866f3615054037972305ea26a259efa995285f30c067e411151d41127a"
Nov 04 00:30:21.317537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.315950    5074 scope.go:122] "RemoveContainer" containerID="d9451b3950b81a1d0a592adcc164c234c6579122b42dc5ffb940806c732c6946"
Nov 04 00:30:21.342914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.342882    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"]
Nov 04 00:30:21.352567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.352435    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895870-cl5z9"]
Nov 04 00:30:21.353130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.353114    5074 scope.go:122] "RemoveContainer" containerID="48d486657bd9065ace9da328975153626fea4d8a2545507f36c38481f5400a44"
Nov 04 00:30:21.362842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.362812    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca\") pod \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") "
Nov 04 00:30:21.362946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.362929    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp\" (UniqueName: \"kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp\") pod \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\" (UID: \"e62769d7-dfe2-4b00-9065-b1a4294bbc13\") "
Nov 04 00:30:21.363145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.363120    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca" pod "e62769d7-dfe2-4b00-9065-b1a4294bbc13" (UID: "e62769d7-dfe2-4b00-9065-b1a4294bbc13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:21.363542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.363513    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62769d7-dfe2-4b00-9065-b1a4294bbc13-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.365732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.365705    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp" pod "e62769d7-dfe2-4b00-9065-b1a4294bbc13" (UID: "e62769d7-dfe2-4b00-9065-b1a4294bbc13"). InnerVolumeSpecName "kube-api-access-9blzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:21.366683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.366661    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"]
Nov 04 00:30:21.377180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.377147    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895870-bv8fd"]
Nov 04 00:30:21.418312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.417764    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:21.464977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.464941    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blzp\" (UniqueName: \"kubernetes.io/projected/e62769d7-dfe2-4b00-9065-b1a4294bbc13-kube-api-access-9blzp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.489982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.489952    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:21.566231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.566060    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca\") pod \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") "
Nov 04 00:30:21.566371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.566333    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh\" (UniqueName: \"kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh\") pod \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\" (UID: \"6d407d9a-345c-495f-9ab2-2f2eb28bf209\") "
Nov 04 00:30:21.566708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.566406    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48\" (UniqueName: \"kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48\") pod \"24b7122f-745a-4828-b1c1-17a3ac6baa04\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") "
Nov 04 00:30:21.566708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.566506    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca" pod "6d407d9a-345c-495f-9ab2-2f2eb28bf209" (UID: "6d407d9a-345c-495f-9ab2-2f2eb28bf209"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:21.566708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.566549    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca\") pod \"24b7122f-745a-4828-b1c1-17a3ac6baa04\" (UID: \"24b7122f-745a-4828-b1c1-17a3ac6baa04\") "
Nov 04 00:30:21.567304 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.567106    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca" pod "24b7122f-745a-4828-b1c1-17a3ac6baa04" (UID: "24b7122f-745a-4828-b1c1-17a3ac6baa04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:21.567304 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.567185    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d407d9a-345c-495f-9ab2-2f2eb28bf209-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.569479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.569452    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh" pod "6d407d9a-345c-495f-9ab2-2f2eb28bf209" (UID: "6d407d9a-345c-495f-9ab2-2f2eb28bf209"). InnerVolumeSpecName "kube-api-access-rpdbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:21.570261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.570236    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48" pod "24b7122f-745a-4828-b1c1-17a3ac6baa04" (UID: "24b7122f-745a-4828-b1c1-17a3ac6baa04"). InnerVolumeSpecName "kube-api-access-kkn48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:21.668843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.668813    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdbh\" (UniqueName: \"kubernetes.io/projected/6d407d9a-345c-495f-9ab2-2f2eb28bf209-kube-api-access-rpdbh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.668843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.668844    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkn48\" (UniqueName: \"kubernetes.io/projected/24b7122f-745a-4828-b1c1-17a3ac6baa04-kube-api-access-kkn48\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.669020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.668858    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b7122f-745a-4828-b1c1-17a3ac6baa04-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.774098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.774068    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:21.778969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.778946    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:21.871391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.871311    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca\") pod \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") "
Nov 04 00:30:21.871588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.871404    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca\") pod \"4c870d06-ff21-4c4e-89fc-884641120889\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") "
Nov 04 00:30:21.871588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.871426    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs\" (UniqueName: \"kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs\") pod \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\" (UID: \"c92c8522-e3a0-4745-a92e-299fc4e30a3a\") "
Nov 04 00:30:21.871588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.871454    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c\" (UniqueName: \"kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c\") pod \"4c870d06-ff21-4c4e-89fc-884641120889\" (UID: \"4c870d06-ff21-4c4e-89fc-884641120889\") "
Nov 04 00:30:21.871802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.871726    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca" pod "c92c8522-e3a0-4745-a92e-299fc4e30a3a" (UID: "c92c8522-e3a0-4745-a92e-299fc4e30a3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:21.871876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.871853    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca" pod "4c870d06-ff21-4c4e-89fc-884641120889" (UID: "4c870d06-ff21-4c4e-89fc-884641120889"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:21.872095 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.872077    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92c8522-e3a0-4745-a92e-299fc4e30a3a-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.872137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.872097    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c870d06-ff21-4c4e-89fc-884641120889-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.873636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.873618    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs" pod "c92c8522-e3a0-4745-a92e-299fc4e30a3a" (UID: "c92c8522-e3a0-4745-a92e-299fc4e30a3a"). InnerVolumeSpecName "kube-api-access-6rjjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:21.873697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.873644    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c" pod "4c870d06-ff21-4c4e-89fc-884641120889" (UID: "4c870d06-ff21-4c4e-89fc-884641120889"). InnerVolumeSpecName "kube-api-access-q2n9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:21.973953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.973914    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rjjs\" (UniqueName: \"kubernetes.io/projected/c92c8522-e3a0-4745-a92e-299fc4e30a3a-kube-api-access-6rjjs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.973953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:21.973954    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2n9c\" (UniqueName: \"kubernetes.io/projected/4c870d06-ff21-4c4e-89fc-884641120889-kube-api-access-q2n9c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.006586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.006548    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"]
Nov 04 00:30:22.025489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.025454    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"]
Nov 04 00:30:22.061197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.061162    5074 generic.go:356] "Generic (PLEG): container finished" podID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerID="dda093c185a5d94172124f61294c16f59e44193ea1eec8a5aa7ed254ca1f8412" exitCode=60
Nov 04 00:30:22.061670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.061236    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" event={"ID":"590353e1-672c-467b-8a8d-ecc4cf08bb55","Type":"ContainerDied","Data":"dda093c185a5d94172124f61294c16f59e44193ea1eec8a5aa7ed254ca1f8412"}
Nov 04 00:30:22.061670 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.061300    5074 scope.go:122] "RemoveContainer" containerID="a9d9d4e385e2b0d4df4f26b65f35a8b2ee42ddfc3df321a8bdfc1fc68b4f07c0"
Nov 04 00:30:22.061901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.061876    5074 scope.go:122] "RemoveContainer" containerID="dda093c185a5d94172124f61294c16f59e44193ea1eec8a5aa7ed254ca1f8412"
Nov 04 00:30:22.062355 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:22.062320    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f_odh-ai-gateway-infra(590353e1-672c-467b-8a8d-ecc4cf08bb55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55"
Nov 04 00:30:22.063811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.063773    5074 generic.go:356] "Generic (PLEG): container finished" podID="11a6425b-0808-4542-8c65-d16448a10baf" containerID="a9d0fd02afe5287a3f332da246ba6e850e703022a0b4000e71871756e01cf4fc" exitCode=60
Nov 04 00:30:22.063811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.063791    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerDied","Data":"a9d0fd02afe5287a3f332da246ba6e850e703022a0b4000e71871756e01cf4fc"}
Nov 04 00:30:22.064184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.064164    5074 scope.go:122] "RemoveContainer" containerID="a9d0fd02afe5287a3f332da246ba6e850e703022a0b4000e71871756e01cf4fc"
Nov 04 00:30:22.064503 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:22.064475    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4_odh-ai-gateway-infra(11a6425b-0808-4542-8c65-d16448a10baf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" podUID="11a6425b-0808-4542-8c65-d16448a10baf"
Nov 04 00:30:22.065313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.065294    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"
Nov 04 00:30:22.065420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.065313    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5" event={"ID":"6d407d9a-345c-495f-9ab2-2f2eb28bf209","Type":"ContainerDied","Data":"ff6a0ea2748480a7219591add79ebc442909fdbfd9bea29973cd3fe9cf280e1b"}
Nov 04 00:30:22.067329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.067301    5074 generic.go:356] "Generic (PLEG): container finished" podID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerID="5cf5dedab7659138fa08e14910708c20a1fd56bca24edbe93c3a79ac8bd431a2" exitCode=60
Nov 04 00:30:22.067434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.067372    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" event={"ID":"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e","Type":"ContainerDied","Data":"5cf5dedab7659138fa08e14910708c20a1fd56bca24edbe93c3a79ac8bd431a2"}
Nov 04 00:30:22.067806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.067775    5074 scope.go:122] "RemoveContainer" containerID="5cf5dedab7659138fa08e14910708c20a1fd56bca24edbe93c3a79ac8bd431a2"
Nov 04 00:30:22.068129 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:22.068041    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn_odh-ai-gateway-infra(5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e"
Nov 04 00:30:22.069860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.069841    5074 generic.go:356] "Generic (PLEG): container finished" podID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerID="c8932ac3999d19557dbaebd0c757dbe5980d1180552232c565c932f543993910" exitCode=60
Nov 04 00:30:22.069982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.069908    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerDied","Data":"c8932ac3999d19557dbaebd0c757dbe5980d1180552232c565c932f543993910"}
Nov 04 00:30:22.070317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.070299    5074 scope.go:122] "RemoveContainer" containerID="c8932ac3999d19557dbaebd0c757dbe5980d1180552232c565c932f543993910"
Nov 04 00:30:22.070709 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:22.070578    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx_odh-ai-gateway-infra(e4961c11-d4e7-4ccd-99e8-13766e5fea74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74"
Nov 04 00:30:22.074490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.074468    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"]
Nov 04 00:30:22.075256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.075227    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"
Nov 04 00:30:22.075356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.075231    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk" event={"ID":"c92c8522-e3a0-4745-a92e-299fc4e30a3a","Type":"ContainerDied","Data":"45ceedb4bd91f5a8c9bfb48625963b4a95e0a5434291453be345504ebc66dc83"}
Nov 04 00:30:22.077878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.077859    5074 generic.go:356] "Generic (PLEG): container finished" podID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerID="fcc57e80e3f6f44308e321540e6a45e21d53af56b18cf93fbee7adc455587824" exitCode=60
Nov 04 00:30:22.078527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.078503    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerDied","Data":"fcc57e80e3f6f44308e321540e6a45e21d53af56b18cf93fbee7adc455587824"}
Nov 04 00:30:22.078877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.078863    5074 scope.go:122] "RemoveContainer" containerID="fcc57e80e3f6f44308e321540e6a45e21d53af56b18cf93fbee7adc455587824"
Nov 04 00:30:22.079072 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:22.079055    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq_odh-ai-gateway-infra(ac6e6048-8127-41d1-b521-4661c5462b8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f"
Nov 04 00:30:22.080592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.080576    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"
Nov 04 00:30:22.080686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.080610    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm" event={"ID":"4c870d06-ff21-4c4e-89fc-884641120889","Type":"ContainerDied","Data":"53d11a7809c6f5dded66454b297a22d9cb84dc39110295cd2527cc87462161ec"}
Nov 04 00:30:22.081854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.081815    5074 scope.go:122] "RemoveContainer" containerID="ac1c3c12409ee56d7253594889c6f910dd62af00e0fb2412127fd3bbd4fe64ca"
Nov 04 00:30:22.082657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.082260    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"
Nov 04 00:30:22.082657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.082300    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr" event={"ID":"24b7122f-745a-4828-b1c1-17a3ac6baa04","Type":"ContainerDied","Data":"21bfc0ca777e9e30ed74e3cd8cc1f7da55dbadc34753b40b13e4b085a3e202f8"}
Nov 04 00:30:22.084074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.084052    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"
Nov 04 00:30:22.084197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.084145    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g" event={"ID":"e62769d7-dfe2-4b00-9065-b1a4294bbc13","Type":"ContainerDied","Data":"93e42579522b9eb648deb1913c87c0ac9e2cadb9a36f72d640b58080693d183a"}
Nov 04 00:30:22.092177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.092151    5074 generic.go:356] "Generic (PLEG): container finished" podID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerID="af0923b82f291fb02d9f393a6e91e1358ae61ce10bbf1828b3f9b5e80339af4b" exitCode=60
Nov 04 00:30:22.092259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.092225    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerDied","Data":"af0923b82f291fb02d9f393a6e91e1358ae61ce10bbf1828b3f9b5e80339af4b"}
Nov 04 00:30:22.094977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.094881    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" event={"ID":"811cb4e3-12f4-4791-9ed1-0807d4045f5e","Type":"ContainerStarted","Data":"f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a"}
Nov 04 00:30:22.107634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.107619    5074 scope.go:122] "RemoveContainer" containerID="7f24e2652e0a8480f4348cd9e0d832d5ce4ea69cbf451a1b4b98ab486e9d0c6b"
Nov 04 00:30:22.108323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.108279    5074 generic.go:356] "Generic (PLEG): container finished" podID="01411085-744b-4c3a-b58b-d995b0497099" containerID="a1027daa4cd47cd1b9c25ef5f0d155e134d9b6047fc5988dce096d255c6a2382" exitCode=60
Nov 04 00:30:22.108499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.108380    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerDied","Data":"a1027daa4cd47cd1b9c25ef5f0d155e134d9b6047fc5988dce096d255c6a2382"}
Nov 04 00:30:22.110929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.110902    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"]
Nov 04 00:30:22.118968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.118123    5074 generic.go:356] "Generic (PLEG): container finished" podID="361534ef-f26c-481e-80da-b758e56df5c0" containerID="c504effeb406f646a337a28342b30f7f494a7dc352cee04afd8fa093b84ef7af" exitCode=60
Nov 04 00:30:22.118968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.118173    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerDied","Data":"c504effeb406f646a337a28342b30f7f494a7dc352cee04afd8fa093b84ef7af"}
Nov 04 00:30:22.118968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.118551    5074 scope.go:122] "RemoveContainer" containerID="c504effeb406f646a337a28342b30f7f494a7dc352cee04afd8fa093b84ef7af"
Nov 04 00:30:22.118968 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:22.118905    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb_odh-ai-gateway-infra(361534ef-f26c-481e-80da-b758e56df5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" podUID="361534ef-f26c-481e-80da-b758e56df5c0"
Nov 04 00:30:22.154416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.154366    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"]
Nov 04 00:30:22.183500 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.183465    5074 scope.go:122] "RemoveContainer" containerID="5c991b58af5c2bda108241bb9a758f29e7a1f75065de5217967439b775c85791"
Nov 04 00:30:22.245012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.244713    5074 scope.go:122] "RemoveContainer" containerID="f831682cc6673ba11ec7a4d013c9272b256cd7f1aa746e4142e6f0b2736048cb"
Nov 04 00:30:22.269199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.269157    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"]
Nov 04 00:30:22.289595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.289566    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"]
Nov 04 00:30:22.331636 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.331588    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a5948-29895870-dfkc5"]
Nov 04 00:30:22.335632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.334057    5074 scope.go:122] "RemoveContainer" containerID="b51fe33700bd66201be6ca22950c58645504be65ddaa749bce39f7310d70594c"
Nov 04 00:30:22.361298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.361206    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"]
Nov 04 00:30:22.383681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.383155    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" podStartSLOduration=3.383136708 podStartE2EDuration="3.383136708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:22.284457438 +0000 UTC m=+3687.384057586" watchObservedRunningTime="2026-11-04 00:30:22.383136708 +0000 UTC m=+3687.482736853"
Nov 04 00:30:22.415886 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.414843    5074 scope.go:122] "RemoveContainer" containerID="05cee743f23344ebda53791091a1c20e12e8d1aaecdb1bfe8d1725537baf99b4"
Nov 04 00:30:22.464968 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.464730    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:22.467449 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.467404    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"]
Nov 04 00:30:22.482845 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.482692    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ade4-29895870-lzdzm"]
Nov 04 00:30:22.502030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.501891    5074 scope.go:122] "RemoveContainer" containerID="dabfee51e63411b31008c836f372c7b947cdc649e9f6a3e7bd659dd1115e6b89"
Nov 04 00:30:22.511433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.511362    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"]
Nov 04 00:30:22.530444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.530365    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895870-ww9hr"]
Nov 04 00:30:22.552860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.552019    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"]
Nov 04 00:30:22.553197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.553078    5074 scope.go:122] "RemoveContainer" containerID="39f609460e5fb2e06c642c94e5d4719643342dc34e737dd61d2e5c8ea2eed1a3"
Nov 04 00:30:22.564233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.564203    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9c0227c-29895870-z7n7g"]
Nov 04 00:30:22.571073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.571023    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"]
Nov 04 00:30:22.586701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.586665    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895870-br7qk"]
Nov 04 00:30:22.599770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.599673    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca\") pod \"f9ba160e-304a-46cd-aad9-4b0151a85b44\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") "
Nov 04 00:30:22.604283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.599752    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv\" (UniqueName: \"kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv\") pod \"f9ba160e-304a-46cd-aad9-4b0151a85b44\" (UID: \"f9ba160e-304a-46cd-aad9-4b0151a85b44\") "
Nov 04 00:30:22.605109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.605071    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca" pod "f9ba160e-304a-46cd-aad9-4b0151a85b44" (UID: "f9ba160e-304a-46cd-aad9-4b0151a85b44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:22.658199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.658158    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:22.669511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.669349    5074 scope.go:122] "RemoveContainer" containerID="c113febadd676042b9f12991ff94faeabfccf346ad2477cb4e58d0bc469320cd"
Nov 04 00:30:22.697941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.697891    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv" pod "f9ba160e-304a-46cd-aad9-4b0151a85b44" (UID: "f9ba160e-304a-46cd-aad9-4b0151a85b44"). InnerVolumeSpecName "kube-api-access-7wkrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:22.701867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.701805    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ba160e-304a-46cd-aad9-4b0151a85b44-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.701867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.701839    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wkrv\" (UniqueName: \"kubernetes.io/projected/f9ba160e-304a-46cd-aad9-4b0151a85b44-kube-api-access-7wkrv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.707251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.706751    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:22.715106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.708659    5074 scope.go:122] "RemoveContainer" containerID="90208ee691c0f8e0cb82433f33e4aba3c2c3c51fb7b76545af6b8796eb1cf70d"
Nov 04 00:30:22.728825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.728800    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:22.745931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.745912    5074 scope.go:122] "RemoveContainer" containerID="3dac93883382ce4636a2c11d04d9a10e5663e1fa32faf7991c65849ef6f5d98c"
Nov 04 00:30:22.747132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.747111    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:22.770526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.770491    5074 scope.go:122] "RemoveContainer" containerID="9efd474ec4cdb8b97e5ab258a603a7be5c49ce337e1848b4b3b7905f3a1af84f"
Nov 04 00:30:22.803508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.803382    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp\" (UniqueName: \"kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp\") pod \"484c335d-2c01-4c31-a93a-528c2361fb96\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") "
Nov 04 00:30:22.803605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.803576    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca\") pod \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") "
Nov 04 00:30:22.803756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.803639    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca\") pod \"01411085-744b-4c3a-b58b-d995b0497099\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") "
Nov 04 00:30:22.803756 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.803718    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk\" (UniqueName: \"kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk\") pod \"01411085-744b-4c3a-b58b-d995b0497099\" (UID: \"01411085-744b-4c3a-b58b-d995b0497099\") "
Nov 04 00:30:22.803883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.803777    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7\" (UniqueName: \"kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7\") pod \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\" (UID: \"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12\") "
Nov 04 00:30:22.803883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.803836    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca\") pod \"484c335d-2c01-4c31-a93a-528c2361fb96\" (UID: \"484c335d-2c01-4c31-a93a-528c2361fb96\") "
Nov 04 00:30:22.805185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.804995    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca" pod "01411085-744b-4c3a-b58b-d995b0497099" (UID: "01411085-744b-4c3a-b58b-d995b0497099"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:22.805349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.805327    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca" pod "b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" (UID: "b3b8bc1c-80fd-4dda-b44c-6996cc16fe12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:22.805751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.805718    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca" pod "484c335d-2c01-4c31-a93a-528c2361fb96" (UID: "484c335d-2c01-4c31-a93a-528c2361fb96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:22.807897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.807858    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp" pod "484c335d-2c01-4c31-a93a-528c2361fb96" (UID: "484c335d-2c01-4c31-a93a-528c2361fb96"). InnerVolumeSpecName "kube-api-access-nc7pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:22.812249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.812222    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk" pod "01411085-744b-4c3a-b58b-d995b0497099" (UID: "01411085-744b-4c3a-b58b-d995b0497099"). InnerVolumeSpecName "kube-api-access-cwzdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:22.812614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.812593    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7" pod "b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" (UID: "b3b8bc1c-80fd-4dda-b44c-6996cc16fe12"). InnerVolumeSpecName "kube-api-access-m8vr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:22.905264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905221    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c\" (UniqueName: \"kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c\") pod \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") "
Nov 04 00:30:22.905440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905357    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca\") pod \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\" (UID: \"a3fd1187-3dac-40d1-9472-3ff3c92d6512\") "
Nov 04 00:30:22.905783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905758    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca" pod "a3fd1187-3dac-40d1-9472-3ff3c92d6512" (UID: "a3fd1187-3dac-40d1-9472-3ff3c92d6512"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:22.905916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905901    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484c335d-2c01-4c31-a93a-528c2361fb96-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.905982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905917    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7pp\" (UniqueName: \"kubernetes.io/projected/484c335d-2c01-4c31-a93a-528c2361fb96-kube-api-access-nc7pp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.905982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905927    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd1187-3dac-40d1-9472-3ff3c92d6512-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.905982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905937    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.905982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905950    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01411085-744b-4c3a-b58b-d995b0497099-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.905982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905965    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzdk\" (UniqueName: \"kubernetes.io/projected/01411085-744b-4c3a-b58b-d995b0497099-kube-api-access-cwzdk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.905982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.905978    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vr7\" (UniqueName: \"kubernetes.io/projected/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12-kube-api-access-m8vr7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:22.907619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:22.907593    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c" pod "a3fd1187-3dac-40d1-9472-3ff3c92d6512" (UID: "a3fd1187-3dac-40d1-9472-3ff3c92d6512"). InnerVolumeSpecName "kube-api-access-48x5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.006984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.006947    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48x5c\" (UniqueName: \"kubernetes.io/projected/a3fd1187-3dac-40d1-9472-3ff3c92d6512-kube-api-access-48x5c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.025484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.025446    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" path="/var/lib/kubelet/pods/24b7122f-745a-4828-b1c1-17a3ac6baa04/volumes"
Nov 04 00:30:23.026543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.026512    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c870d06-ff21-4c4e-89fc-884641120889" path="/var/lib/kubelet/pods/4c870d06-ff21-4c4e-89fc-884641120889/volumes"
Nov 04 00:30:23.027188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.027159    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" path="/var/lib/kubelet/pods/6d407d9a-345c-495f-9ab2-2f2eb28bf209/volumes"
Nov 04 00:30:23.028214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.028191    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" path="/var/lib/kubelet/pods/a2284a42-2ab9-4469-b6c4-60a7e18b6ab2/volumes"
Nov 04 00:30:23.028631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.028615    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" path="/var/lib/kubelet/pods/b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3/volumes"
Nov 04 00:30:23.029060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.029025    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" path="/var/lib/kubelet/pods/c92c8522-e3a0-4745-a92e-299fc4e30a3a/volumes"
Nov 04 00:30:23.029918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.029899    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" path="/var/lib/kubelet/pods/e62769d7-dfe2-4b00-9065-b1a4294bbc13/volumes"
Nov 04 00:30:23.050075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.047395    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:23.099512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.099475    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"]
Nov 04 00:30:23.135205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.135160    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"]
Nov 04 00:30:23.136398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.136376    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"
Nov 04 00:30:23.136398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.136382    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk" event={"ID":"f9ba160e-304a-46cd-aad9-4b0151a85b44","Type":"ContainerDied","Data":"5efad237e954f9ca354db3b30a133d9cb19bb613c6701bc1f5acf9c8206be442"}
Nov 04 00:30:23.136598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.136445    5074 scope.go:122] "RemoveContainer" containerID="a552ada02007941e220f4cc7718af9d3fb5b9df1417aea0a874934d9f3ab6679"
Nov 04 00:30:23.138807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.138348    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"
Nov 04 00:30:23.138807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.138390    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c" event={"ID":"b3b8bc1c-80fd-4dda-b44c-6996cc16fe12","Type":"ContainerDied","Data":"1dd60fbae43b950e4259adda23ba8645e846ed49781d5093f310c2aa3a2535e4"}
Nov 04 00:30:23.141340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.141321    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"
Nov 04 00:30:23.141451 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.141371    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw" event={"ID":"01411085-744b-4c3a-b58b-d995b0497099","Type":"ContainerDied","Data":"680c1849cf43c862724635210f46bf34c4bfcf524e4e5d7f41988da4f6d5f43b"}
Nov 04 00:30:23.147219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.147200    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"
Nov 04 00:30:23.147600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.147229    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl" event={"ID":"484c335d-2c01-4c31-a93a-528c2361fb96","Type":"ContainerDied","Data":"6fc6031abbfc8c69127674aa757c92d5ac5df19729c414024d0e7014623af0b6"}
Nov 04 00:30:23.160892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.160624    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx" event={"ID":"a3fd1187-3dac-40d1-9472-3ff3c92d6512","Type":"ContainerDied","Data":"0217bc2ebef457e1f0541936b20581b6459cf6ab5fc2fabdef5388fadfebae63"}
Nov 04 00:30:23.160892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.160726    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"
Nov 04 00:30:23.240951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.240916    5074 scope.go:122] "RemoveContainer" containerID="af0923b82f291fb02d9f393a6e91e1358ae61ce10bbf1828b3f9b5e80339af4b"
Nov 04 00:30:23.405329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.405296    5074 scope.go:122] "RemoveContainer" containerID="a1027daa4cd47cd1b9c25ef5f0d155e134d9b6047fc5988dce096d255c6a2382"
Nov 04 00:30:23.434444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.430567    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"]
Nov 04 00:30:23.460946 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.459931    5074 scope.go:122] "RemoveContainer" containerID="48808a70371ea504d3ffc0f89f6bd7789cb8452e602ff731cf4605236dd5bd38"
Nov 04 00:30:23.473446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.470702    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"]
Nov 04 00:30:23.524939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.524868    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"]
Nov 04 00:30:23.527650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.527618    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b3bea7-29895870-jwr7c"]
Nov 04 00:30:23.534065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.533201    5074 scope.go:122] "RemoveContainer" containerID="79f4019f8fb156ab85e001cea87e883a6df83b0d2f51eec9af184bbe245a970f"
Nov 04 00:30:23.564695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.538831    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"]
Nov 04 00:30:23.578177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.576879    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8093f5-29895870-xjnmx"]
Nov 04 00:30:23.578177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.576931    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"]
Nov 04 00:30:23.585545 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.585523    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:23.592927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.592893    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-86deca-29895870-8kprk"]
Nov 04 00:30:23.601869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.601838    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"]
Nov 04 00:30:23.602321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.602301    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a002b-29895870-7vsbw"]
Nov 04 00:30:23.608977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.608948    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"]
Nov 04 00:30:23.612938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.612915    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-80f8ba-29895870-s7tzl"]
Nov 04 00:30:23.664786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.664153    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn\" (UniqueName: \"kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn\") pod \"361534ef-f26c-481e-80da-b758e56df5c0\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") "
Nov 04 00:30:23.675161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.674071    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn" pod "361534ef-f26c-481e-80da-b758e56df5c0" (UID: "361534ef-f26c-481e-80da-b758e56df5c0"). InnerVolumeSpecName "kube-api-access-rhjrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.757046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.755935    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:23.758333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.758303    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:23.761459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.761429    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:23.766320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.766291    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m\" (UniqueName: \"kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m\") pod \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") "
Nov 04 00:30:23.766878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.766786    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca\") pod \"361534ef-f26c-481e-80da-b758e56df5c0\" (UID: \"361534ef-f26c-481e-80da-b758e56df5c0\") "
Nov 04 00:30:23.767163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.767139    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca" pod "361534ef-f26c-481e-80da-b758e56df5c0" (UID: "361534ef-f26c-481e-80da-b758e56df5c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:23.767286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.767268    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca\") pod \"ac6e6048-8127-41d1-b521-4661c5462b8f\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") "
Nov 04 00:30:23.767870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.767360    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca\") pod \"590353e1-672c-467b-8a8d-ecc4cf08bb55\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") "
Nov 04 00:30:23.768132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.767701    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca" pod "ac6e6048-8127-41d1-b521-4661c5462b8f" (UID: "ac6e6048-8127-41d1-b521-4661c5462b8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:23.768132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.767813    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca" pod "590353e1-672c-467b-8a8d-ecc4cf08bb55" (UID: "590353e1-672c-467b-8a8d-ecc4cf08bb55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:23.768595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.768555    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj\" (UniqueName: \"kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj\") pod \"ac6e6048-8127-41d1-b521-4661c5462b8f\" (UID: \"ac6e6048-8127-41d1-b521-4661c5462b8f\") "
Nov 04 00:30:23.769001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.768966    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v\" (UniqueName: \"kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v\") pod \"590353e1-672c-467b-8a8d-ecc4cf08bb55\" (UID: \"590353e1-672c-467b-8a8d-ecc4cf08bb55\") "
Nov 04 00:30:23.769633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.769613    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca\") pod \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\" (UID: \"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e\") "
Nov 04 00:30:23.770464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.770212    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca" pod "5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" (UID: "5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:23.770895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.770875    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj" pod "ac6e6048-8127-41d1-b521-4661c5462b8f" (UID: "ac6e6048-8127-41d1-b521-4661c5462b8f"). InnerVolumeSpecName "kube-api-access-w78pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.771298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.771270    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361534ef-f26c-481e-80da-b758e56df5c0-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.771377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.771306    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6e6048-8127-41d1-b521-4661c5462b8f-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.771377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.771321    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhjrn\" (UniqueName: \"kubernetes.io/projected/361534ef-f26c-481e-80da-b758e56df5c0-kube-api-access-rhjrn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.771377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.771336    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/590353e1-672c-467b-8a8d-ecc4cf08bb55-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.771377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.771350    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w78pj\" (UniqueName: \"kubernetes.io/projected/ac6e6048-8127-41d1-b521-4661c5462b8f-kube-api-access-w78pj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.771377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.771365    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.798520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.798310    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m" pod "5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" (UID: "5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e"). InnerVolumeSpecName "kube-api-access-7tz4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.798520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.798369    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v" pod "590353e1-672c-467b-8a8d-ecc4cf08bb55" (UID: "590353e1-672c-467b-8a8d-ecc4cf08bb55"). InnerVolumeSpecName "kube-api-access-62w6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.813324 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.812897    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:23.827172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.827028    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:23.846953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.846863    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"]
Nov 04 00:30:23.847331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.847094    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" podUID="0879e1e4-f187-4442-bc2f-35b90eb8931a" containerName="maas-api" containerID="cri-o://726b8dd9474e96762dc7e5db8709419f6fa007362dfeea3711c71d33f657d186" gracePeriod=30
Nov 04 00:30:23.872002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.871972    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9\" (UniqueName: \"kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9\") pod \"11a6425b-0808-4542-8c65-d16448a10baf\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") "
Nov 04 00:30:23.872186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.872016    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca\") pod \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") "
Nov 04 00:30:23.872252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.872237    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca\") pod \"11a6425b-0808-4542-8c65-d16448a10baf\" (UID: \"11a6425b-0808-4542-8c65-d16448a10baf\") "
Nov 04 00:30:23.872325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.872304    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5\" (UniqueName: \"kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5\") pod \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\" (UID: \"e4961c11-d4e7-4ccd-99e8-13766e5fea74\") "
Nov 04 00:30:23.872723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.872704    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62w6v\" (UniqueName: \"kubernetes.io/projected/590353e1-672c-467b-8a8d-ecc4cf08bb55-kube-api-access-62w6v\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.872866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.872727    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tz4m\" (UniqueName: \"kubernetes.io/projected/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e-kube-api-access-7tz4m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.873558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.873528    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca" pod "11a6425b-0808-4542-8c65-d16448a10baf" (UID: "11a6425b-0808-4542-8c65-d16448a10baf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:23.874056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.874027    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca" pod "e4961c11-d4e7-4ccd-99e8-13766e5fea74" (UID: "e4961c11-d4e7-4ccd-99e8-13766e5fea74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:23.877199 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.877169    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5" pod "e4961c11-d4e7-4ccd-99e8-13766e5fea74" (UID: "e4961c11-d4e7-4ccd-99e8-13766e5fea74"). InnerVolumeSpecName "kube-api-access-cqkc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.878162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.878138    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9" pod "11a6425b-0808-4542-8c65-d16448a10baf" (UID: "11a6425b-0808-4542-8c65-d16448a10baf"). InnerVolumeSpecName "kube-api-access-72fj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:23.973758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.973707    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4961c11-d4e7-4ccd-99e8-13766e5fea74-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.973959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.973787    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a6425b-0808-4542-8c65-d16448a10baf-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.973959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.973802    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkc5\" (UniqueName: \"kubernetes.io/projected/e4961c11-d4e7-4ccd-99e8-13766e5fea74-kube-api-access-cqkc5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:23.973959 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:23.973812    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fj9\" (UniqueName: \"kubernetes.io/projected/11a6425b-0808-4542-8c65-d16448a10baf-kube-api-access-72fj9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:24.271112 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.271084    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"
Nov 04 00:30:24.271714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.271689    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq" event={"ID":"ac6e6048-8127-41d1-b521-4661c5462b8f","Type":"ContainerDied","Data":"749b3675fed3622ee493129901318d24883a8f9612d92d7918e7ee60fd709076"}
Nov 04 00:30:24.271847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.271784    5074 scope.go:122] "RemoveContainer" containerID="fcc57e80e3f6f44308e321540e6a45e21d53af56b18cf93fbee7adc455587824"
Nov 04 00:30:24.277948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.277919    5074 generic.go:356] "Generic (PLEG): container finished" podID="0879e1e4-f187-4442-bc2f-35b90eb8931a" containerID="726b8dd9474e96762dc7e5db8709419f6fa007362dfeea3711c71d33f657d186" exitCode=0
Nov 04 00:30:24.278051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.278009    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" event={"ID":"0879e1e4-f187-4442-bc2f-35b90eb8931a","Type":"ContainerDied","Data":"726b8dd9474e96762dc7e5db8709419f6fa007362dfeea3711c71d33f657d186"}
Nov 04 00:30:24.281970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.281909    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"
Nov 04 00:30:24.281970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.281939    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb" event={"ID":"361534ef-f26c-481e-80da-b758e56df5c0","Type":"ContainerDied","Data":"de6d6a8d9542ac773c5b6521af2c86e5d6c4d93c3515aeb7582a311c00480be1"}
Nov 04 00:30:24.285733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.285705    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f" event={"ID":"590353e1-672c-467b-8a8d-ecc4cf08bb55","Type":"ContainerDied","Data":"1daf59f0bb1b4386d66d6ff992e26cf10066c0da61e8eab2323b6f9b348c44ad"}
Nov 04 00:30:24.285931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.285826    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"
Nov 04 00:30:24.292435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.292369    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"
Nov 04 00:30:24.292618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.292385    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4" event={"ID":"11a6425b-0808-4542-8c65-d16448a10baf","Type":"ContainerDied","Data":"98cc20b9edb36923a0c5f2e1761e528538368eca48e9c02b48f145a74030d8fd"}
Nov 04 00:30:24.294158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.294122    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn" event={"ID":"5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e","Type":"ContainerDied","Data":"78e2a53aeb761976ea99c0b25aef83f89ab18cba0383cf8a1b6e55ccc05fa810"}
Nov 04 00:30:24.296358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.296015    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx" event={"ID":"e4961c11-d4e7-4ccd-99e8-13766e5fea74","Type":"ContainerDied","Data":"1fd3c6d58fda6f237c1f937b5c48209d47fee15092b34f5e48aced4e163a9a1a"}
Nov 04 00:30:24.297486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.297469    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"
Nov 04 00:30:24.298785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.298552    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"
Nov 04 00:30:24.321383 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.319107    5074 scope.go:122] "RemoveContainer" containerID="c504effeb406f646a337a28342b30f7f494a7dc352cee04afd8fa093b84ef7af"
Nov 04 00:30:24.327262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.327191    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:30:24.341821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.341796    5074 scope.go:122] "RemoveContainer" containerID="dda093c185a5d94172124f61294c16f59e44193ea1eec8a5aa7ed254ca1f8412"
Nov 04 00:30:24.346260 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.346236    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"]
Nov 04 00:30:24.351553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.351532    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-71b2f3-29895870-xflxb"]
Nov 04 00:30:24.355365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.355329    5074 scope.go:122] "RemoveContainer" containerID="a9d0fd02afe5287a3f332da246ba6e850e703022a0b4000e71871756e01cf4fc"
Nov 04 00:30:24.364282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.364260    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"]
Nov 04 00:30:24.367939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.367919    5074 scope.go:122] "RemoveContainer" containerID="5cf5dedab7659138fa08e14910708c20a1fd56bca24edbe93c3a79ac8bd431a2"
Nov 04 00:30:24.368960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.368906    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895870-vwbrn"]
Nov 04 00:30:24.378347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.378328    5074 scope.go:122] "RemoveContainer" containerID="c8932ac3999d19557dbaebd0c757dbe5980d1180552232c565c932f543993910"
Nov 04 00:30:24.381929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.381910    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"]
Nov 04 00:30:24.387498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.387469    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895870-hq86f"]
Nov 04 00:30:24.406952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.406930    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"]
Nov 04 00:30:24.410733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.410713    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8895cb-29895870-d7xrq"]
Nov 04 00:30:24.428713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.428685    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca\") pod \"0879e1e4-f187-4442-bc2f-35b90eb8931a\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") "
Nov 04 00:30:24.428863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.428833    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\") pod \"0879e1e4-f187-4442-bc2f-35b90eb8931a\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") "
Nov 04 00:30:24.428939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.428910    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs\") pod \"0879e1e4-f187-4442-bc2f-35b90eb8931a\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") "
Nov 04 00:30:24.429054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.429038    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk\" (UniqueName: \"kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk\") pod \"0879e1e4-f187-4442-bc2f-35b90eb8931a\" (UID: \"0879e1e4-f187-4442-bc2f-35b90eb8931a\") "
Nov 04 00:30:24.430187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.430161    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca" pod "0879e1e4-f187-4442-bc2f-35b90eb8931a" (UID: "0879e1e4-f187-4442-bc2f-35b90eb8931a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:24.435248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.435228    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs" pod "0879e1e4-f187-4442-bc2f-35b90eb8931a" (UID: "0879e1e4-f187-4442-bc2f-35b90eb8931a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:30:24.435409 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.435386    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls" pod "0879e1e4-f187-4442-bc2f-35b90eb8931a" (UID: "0879e1e4-f187-4442-bc2f-35b90eb8931a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:30:24.435855 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.435828    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk" pod "0879e1e4-f187-4442-bc2f-35b90eb8931a" (UID: "0879e1e4-f187-4442-bc2f-35b90eb8931a"). InnerVolumeSpecName "kube-api-access-rtdrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:24.466105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.466072    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"]
Nov 04 00:30:24.469590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.469558    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895870-d2prx"]
Nov 04 00:30:24.478819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.478789    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"]
Nov 04 00:30:24.488474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.488445    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4376d-29895870-zd6n4"]
Nov 04 00:30:24.530576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.530547    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdrk\" (UniqueName: \"kubernetes.io/projected/0879e1e4-f187-4442-bc2f-35b90eb8931a-kube-api-access-rtdrk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:24.530576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.530577    5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0879e1e4-f187-4442-bc2f-35b90eb8931a-openshift-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:24.530723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.530591    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-maas-api-tls\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:24.530723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:24.530600    5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0879e1e4-f187-4442-bc2f-35b90eb8931a-metrics-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:25.022619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.022577    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01411085-744b-4c3a-b58b-d995b0497099" path="/var/lib/kubelet/pods/01411085-744b-4c3a-b58b-d995b0497099/volumes"
Nov 04 00:30:25.023129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.023111    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a6425b-0808-4542-8c65-d16448a10baf" path="/var/lib/kubelet/pods/11a6425b-0808-4542-8c65-d16448a10baf/volumes"
Nov 04 00:30:25.023560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.023544    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361534ef-f26c-481e-80da-b758e56df5c0" path="/var/lib/kubelet/pods/361534ef-f26c-481e-80da-b758e56df5c0/volumes"
Nov 04 00:30:25.024325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.024308    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" path="/var/lib/kubelet/pods/484c335d-2c01-4c31-a93a-528c2361fb96/volumes"
Nov 04 00:30:25.024714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.024700    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" path="/var/lib/kubelet/pods/590353e1-672c-467b-8a8d-ecc4cf08bb55/volumes"
Nov 04 00:30:25.025132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.025116    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" path="/var/lib/kubelet/pods/5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e/volumes"
Nov 04 00:30:25.025803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.025784    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" path="/var/lib/kubelet/pods/a3fd1187-3dac-40d1-9472-3ff3c92d6512/volumes"
Nov 04 00:30:25.026247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.026230    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" path="/var/lib/kubelet/pods/ac6e6048-8127-41d1-b521-4661c5462b8f/volumes"
Nov 04 00:30:25.026626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.026611    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" path="/var/lib/kubelet/pods/b3b8bc1c-80fd-4dda-b44c-6996cc16fe12/volumes"
Nov 04 00:30:25.027616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.027594    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" path="/var/lib/kubelet/pods/e4961c11-d4e7-4ccd-99e8-13766e5fea74/volumes"
Nov 04 00:30:25.028121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.028102    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" path="/var/lib/kubelet/pods/f9ba160e-304a-46cd-aad9-4b0151a85b44/volumes"
Nov 04 00:30:25.311518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.311434    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"
Nov 04 00:30:25.311518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.311461    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5" event={"ID":"0879e1e4-f187-4442-bc2f-35b90eb8931a","Type":"ContainerDied","Data":"1bdee0b9790045485a0349ddd08cddffb4d5264547e837832302f292d233402d"}
Nov 04 00:30:25.312039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.311530    5074 scope.go:122] "RemoveContainer" containerID="726b8dd9474e96762dc7e5db8709419f6fa007362dfeea3711c71d33f657d186"
Nov 04 00:30:25.333965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.333937    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"]
Nov 04 00:30:25.337820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:25.337792    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9c0227c-6699cb8c64-5frl5"]
Nov 04 00:30:27.026314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:27.026274    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0879e1e4-f187-4442-bc2f-35b90eb8931a" path="/var/lib/kubelet/pods/0879e1e4-f187-4442-bc2f-35b90eb8931a/volumes"
Nov 04 00:30:28.024854 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:28.024815    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:30:30.040472 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:30.040440    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:30.053395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:30.053367    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:30.378588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:30.378490    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:30:32.779320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.779278    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"]
Nov 04 00:30:32.780142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780110    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:30:32.780142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780130    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:30:32.780142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780140    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.780142 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780146    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780152    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780160    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780169    5074 cpu_manager.go:414] "containerMap: removing container" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780173    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780182    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780186    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780195    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780200    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780209    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780213    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780219    5074 cpu_manager.go:414] "containerMap: removing container" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780224    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780233    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780238    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780246    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780251    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780261    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780265    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780271    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780275    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780281    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780285    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780291    5074 cpu_manager.go:414] "containerMap: removing container" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780296    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780302    5074 cpu_manager.go:414] "containerMap: removing container" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780306    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780312    5074 cpu_manager.go:414] "containerMap: removing container" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780316    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780324    5074 cpu_manager.go:414] "containerMap: removing container" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780329    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780337    5074 cpu_manager.go:414] "containerMap: removing container" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780341    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780351    5074 cpu_manager.go:414] "containerMap: removing container" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780355    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780366    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0879e1e4-f187-4442-bc2f-35b90eb8931a" containerName="maas-api"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780371    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0879e1e4-f187-4442-bc2f-35b90eb8931a" containerName="maas-api"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780378    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780382    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780388    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780392    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780400    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780406    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780415    5074 cpu_manager.go:414] "containerMap: removing container" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780420    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780427    5074 cpu_manager.go:414] "containerMap: removing container" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780432    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780439    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780443    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780449    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:30:32.780421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780454    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780460    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780465    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780471    5074 cpu_manager.go:414] "containerMap: removing container" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780475    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780482    5074 cpu_manager.go:414] "containerMap: removing container" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780486    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780493    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780498    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780503    5074 cpu_manager.go:414] "containerMap: removing container" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780508    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780515    5074 cpu_manager.go:414] "containerMap: removing container" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780519    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780526    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780531    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780539    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780544    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780551    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780556    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780562    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780566    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780572    5074 cpu_manager.go:414] "containerMap: removing container" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780576    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780583    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780588    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780593    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780598    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780607    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780610    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780619    5074 cpu_manager.go:414] "containerMap: removing container" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780625    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780634    5074 cpu_manager.go:414] "containerMap: removing container" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780639    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780646    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780651    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780659    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780663    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780672    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780677    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780797    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780805    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780813    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780820    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780826    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780832    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780836    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780842    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780847    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780853    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780858    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780863    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780869    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780873    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780880    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780885    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780889    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780895    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780900    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780906    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780912    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d407d9a-345c-495f-9ab2-2f2eb28bf209" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780918    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780923    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780930    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780937    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780944    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780950    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0879e1e4-f187-4442-bc2f-35b90eb8931a" containerName="maas-api"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780957    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780963    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780969    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780974    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780980    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780987    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.780995    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781000    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781005    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781009    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781016    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="01411085-744b-4c3a-b58b-d995b0497099" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781022    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781027    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781033    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781039    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b8bc1c-80fd-4dda-b44c-6996cc16fe12" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781047    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781055    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:30:32.782927 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.781062    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:30:32.789678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.789653    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.801904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.794249    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"]
Nov 04 00:30:32.801904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.794495    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"default-dockercfg-9h4s2\""
Nov 04 00:30:32.801904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.795807    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"openshift-service-ca.crt\""
Nov 04 00:30:32.801904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.795905    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"kube-root-ca.crt\""
Nov 04 00:30:32.828575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.826901    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-58b75d\"/\"rate-test-model-58b75d-kserve-self-signed-certs\""
Nov 04 00:30:32.847245 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.847216    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.847436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.847268    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.847436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.847322    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.847610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.847462    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.847610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.847531    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.847610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.847567    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xzz\" (UniqueName: \"kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949029 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.948987    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.949081    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.949119    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xzz\" (UniqueName: \"kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.949233    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.949270    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.949322    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.949613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.949573    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.950207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.950180    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.950351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.950253    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.954216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.954183    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.959816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.955708    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:32.986046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:32.985968    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xzz\" (UniqueName: \"kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz\") pod \"rate-test-model-58b75d-kserve-5689484845-9pgrj\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") " pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:33.131370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:33.131265    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:33.426065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:33.426034    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"]
Nov 04 00:30:33.426302 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:30:33.426272    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33318433_8255_441e_b24a_9aa8d1abca1e.slice/crio-acf96e0f77a288b364a56eb55421c28c0c32325ae77479c70e46786337693485 WatchSource:0}: Error finding container acf96e0f77a288b364a56eb55421c28c0c32325ae77479c70e46786337693485: Status 404 returned error can't find the container with id acf96e0f77a288b364a56eb55421c28c0c32325ae77479c70e46786337693485
Nov 04 00:30:34.412228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:34.412191    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj" event={"ID":"33318433-8255-441e-b24a-9aa8d1abca1e","Type":"ContainerStarted","Data":"c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25"}
Nov 04 00:30:34.412820 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:34.412781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj" event={"ID":"33318433-8255-441e-b24a-9aa8d1abca1e","Type":"ContainerStarted","Data":"acf96e0f77a288b364a56eb55421c28c0c32325ae77479c70e46786337693485"}
Nov 04 00:30:34.438395 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:34.438331    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj" podStartSLOduration=2.438311327 podStartE2EDuration="2.438311327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:34.432834228 +0000 UTC m=+3699.532434376" watchObservedRunningTime="2026-11-04 00:30:34.438311327 +0000 UTC m=+3699.537911454"
Nov 04 00:30:35.136759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:35.132061    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:43.131712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:43.131672    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:43.145391 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:43.145355    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:43.525646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:43.525611    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:30:51.304764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:51.304513    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"]
Nov 04 00:30:51.393683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:51.386950    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"]
Nov 04 00:30:56.305517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.305474    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn" podUID="17fcde93-4243-4629-93cc-7a58fbe489ec" containerName="payload-processing" containerID="cri-o://aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d" gracePeriod=25
Nov 04 00:30:56.387772 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.387718    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" podUID="e90267b5-38ea-499b-aa99-2eb911054653" containerName="payload-pre-processing" containerID="cri-o://7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460" gracePeriod=25
Nov 04 00:30:56.610183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.610154    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:30:56.649587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.649551    5074 generic.go:356] "Generic (PLEG): container finished" podID="17fcde93-4243-4629-93cc-7a58fbe489ec" containerID="aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d" exitCode=0
Nov 04 00:30:56.649825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.649644    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"
Nov 04 00:30:56.649825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.649657    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn" event={"ID":"17fcde93-4243-4629-93cc-7a58fbe489ec","Type":"ContainerDied","Data":"aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d"}
Nov 04 00:30:56.649976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.649840    5074 scope.go:122] "RemoveContainer" containerID="aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d"
Nov 04 00:30:56.650174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.649814    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn" event={"ID":"17fcde93-4243-4629-93cc-7a58fbe489ec","Type":"ContainerDied","Data":"6d61202dfcc6e1e04951bdf5a6a22bef76c519fc32d807b51d0b16df838ee4e5"}
Nov 04 00:30:56.664419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.664394    5074 scope.go:122] "RemoveContainer" containerID="aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d"
Nov 04 00:30:56.664753 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:56.664706    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d\": container with ID starting with aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d not found: ID does not exist" containerID="aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d"
Nov 04 00:30:56.664857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.664758    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d"} err="failed to get container status \"aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d\": rpc error: code = NotFound desc = could not find container \"aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d\": container with ID starting with aa9bb00327f2d636d4cf2c99506d54200079b766064a4e5c2f0e791e2f539e9d not found: ID does not exist"
Nov 04 00:30:56.710039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.709988    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume\") pod \"17fcde93-4243-4629-93cc-7a58fbe489ec\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") "
Nov 04 00:30:56.710219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.710080    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck\" (UniqueName: \"kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck\") pod \"17fcde93-4243-4629-93cc-7a58fbe489ec\" (UID: \"17fcde93-4243-4629-93cc-7a58fbe489ec\") "
Nov 04 00:30:56.711871 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.711838    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume" pod "17fcde93-4243-4629-93cc-7a58fbe489ec" (UID: "17fcde93-4243-4629-93cc-7a58fbe489ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:56.714122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.714094    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck" pod "17fcde93-4243-4629-93cc-7a58fbe489ec" (UID: "17fcde93-4243-4629-93cc-7a58fbe489ec"). InnerVolumeSpecName "kube-api-access-htmck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:56.811901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.811817    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmck\" (UniqueName: \"kubernetes.io/projected/17fcde93-4243-4629-93cc-7a58fbe489ec-kube-api-access-htmck\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:56.811901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.811852    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fcde93-4243-4629-93cc-7a58fbe489ec-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:56.978244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:56.978210    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"]
Nov 04 00:30:57.077866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:57.077780    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a002b-5cf7ddf4fb-fsvjn"]
Nov 04 00:30:57.329707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:57.329618    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"]
Nov 04 00:30:57.330198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:57.329928    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy" containerID="cri-o://0fabca77f932d902cb63f9adbc9c1c5f73a51001be06db8477f3467ecf4a9601" gracePeriod=30
Nov 04 00:30:57.397760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:57.397705    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"]
Nov 04 00:30:57.398025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:57.397980    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy" containerID="cri-o://db72f39c672b78c740140110f7129a0589f2e4365e05973dc587dc2caf90c872" gracePeriod=30
Nov 04 00:30:58.025258 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:30:58.025217    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:30:59.024752 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:59.024706    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fcde93-4243-4629-93cc-7a58fbe489ec" path="/var/lib/kubelet/pods/17fcde93-4243-4629-93cc-7a58fbe489ec/volumes"
Nov 04 00:30:59.793481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:30:59.790497    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" podUID="e90267b5-38ea-499b-aa99-2eb911054653" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:31:00.967732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:00.967693    5074 scope.go:122] "RemoveContainer" containerID="a851193a6bde8a6c05d005515b412b8ec036d82ad02ca4fca8e056952f9c5cfd"
Nov 04 00:31:00.980346 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:00.980317    5074 scope.go:122] "RemoveContainer" containerID="67206a1e5103ecf71faa373c95b1858193fb485b52e40b2de40ff6624cc312f1"
Nov 04 00:31:00.990757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:00.990723    5074 scope.go:122] "RemoveContainer" containerID="ceda4767db20d01fb6a50e8423f038faf2d5be1868ddb3a20be92d488b236911"
Nov 04 00:31:01.002556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.002535    5074 scope.go:122] "RemoveContainer" containerID="c5183a470cd0fde58034653bd3bb8b9c1749204c5b8ad3938d425a685e248a2d"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.295287    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"]
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296423    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296447    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296469    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296477    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92c8522-e3a0-4745-a92e-299fc4e30a3a" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296488    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296496    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296519    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296527    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296542    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296550    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296574    5074 cpu_manager.go:414] "containerMap: removing container" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296581    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361534ef-f26c-481e-80da-b758e56df5c0" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296607    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296615    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6e6048-8127-41d1-b521-4661c5462b8f" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296640    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296647    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df5daf5-5414-4e7a-b8fb-2694f0a3ad1e" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296670    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296677    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4961c11-d4e7-4ccd-99e8-13766e5fea74" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296697    5074 cpu_manager.go:414] "containerMap: removing container" podUID="17fcde93-4243-4629-93cc-7a58fbe489ec" containerName="payload-processing"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296705    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fcde93-4243-4629-93cc-7a58fbe489ec" containerName="payload-processing"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296906    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="484c335d-2c01-4c31-a93a-528c2361fb96" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296925    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fcde93-4243-4629-93cc-7a58fbe489ec" containerName="payload-processing"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296938    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="590353e1-672c-467b-8a8d-ecc4cf08bb55" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296952    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fd1187-3dac-40d1-9472-3ff3c92d6512" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296962    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ba160e-304a-46cd-aad9-4b0151a85b44" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296976    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c870d06-ff21-4c4e-89fc-884641120889" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.296988    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96a4d5e-43f2-4bbf-a3bb-483a0ab53ad3" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.297001    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a6425b-0808-4542-8c65-d16448a10baf" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.297014    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62769d7-dfe2-4b00-9065-b1a4294bbc13" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.297025    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b7122f-745a-4828-b1c1-17a3ac6baa04" containerName="cleanup"
Nov 04 00:31:01.297862 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.297040    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2284a42-2ab9-4469-b6c4-60a7e18b6ab2" containerName="cleanup"
Nov 04 00:31:01.320565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.320535    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.323641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.323611    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"]
Nov 04 00:31:01.339812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.332954    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"default-dockercfg-qtzvl\""
Nov 04 00:31:01.339812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.332971    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"kube-root-ca.crt\""
Nov 04 00:31:01.339812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.333153    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"openshift-service-ca.crt\""
Nov 04 00:31:01.358794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.358752    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-8093f5\"/\"rate-test-model-8093f5-kserve-self-signed-certs\""
Nov 04 00:31:01.501762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.498333    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.501762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.498416    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.501762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.498464    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.501762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.498545    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.501762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.498607    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkm8s\" (UniqueName: \"kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.501762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.498727    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.623972    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.624059    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.624146    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.624220    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkm8s\" (UniqueName: \"kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.624342    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.624447    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.625539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.625115    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.667762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.636220    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.667762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.636489    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.704049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.704001    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.720071 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.720021    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.727759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.722501    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkm8s\" (UniqueName: \"kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s\") pod \"rate-test-model-8093f5-kserve-7c9cf68557-rwtfh\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") " pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:01.949176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:01.949140    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:02.246384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.246356    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"]
Nov 04 00:31:02.248285 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:31:02.248255    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e436f23_780c_4727_8e27_9120f9d533f8.slice/crio-dbe4a68b58b068f68cbcd7be5a9644fd53c49bfd4e567100fbb15d848fabe74f WatchSource:0}: Error finding container dbe4a68b58b068f68cbcd7be5a9644fd53c49bfd4e567100fbb15d848fabe74f: Status 404 returned error can't find the container with id dbe4a68b58b068f68cbcd7be5a9644fd53c49bfd4e567100fbb15d848fabe74f
Nov 04 00:31:02.761535 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.761503    5074 generic.go:356] "Generic (PLEG): container finished" podID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerID="0fabca77f932d902cb63f9adbc9c1c5f73a51001be06db8477f3467ecf4a9601" exitCode=0
Nov 04 00:31:02.762546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.762513    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" event={"ID":"951a1a46-d504-4b34-9f6b-12d2a57ba342","Type":"ContainerDied","Data":"0fabca77f932d902cb63f9adbc9c1c5f73a51001be06db8477f3467ecf4a9601"}
Nov 04 00:31:02.765914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.765885    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh" event={"ID":"2e436f23-780c-4727-8e27-9120f9d533f8","Type":"ContainerStarted","Data":"8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44"}
Nov 04 00:31:02.766051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.765926    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh" event={"ID":"2e436f23-780c-4727-8e27-9120f9d533f8","Type":"ContainerStarted","Data":"dbe4a68b58b068f68cbcd7be5a9644fd53c49bfd4e567100fbb15d848fabe74f"}
Nov 04 00:31:02.768296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.768268    5074 generic.go:356] "Generic (PLEG): container finished" podID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerID="db72f39c672b78c740140110f7129a0589f2e4365e05973dc587dc2caf90c872" exitCode=0
Nov 04 00:31:02.768432 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.768360    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" event={"ID":"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33","Type":"ContainerDied","Data":"db72f39c672b78c740140110f7129a0589f2e4365e05973dc587dc2caf90c872"}
Nov 04 00:31:02.801899 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.801816    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh" podStartSLOduration=1.801796102 podStartE2EDuration="1.801796102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:31:02.796984517 +0000 UTC m=+3727.896584664" watchObservedRunningTime="2026-11-04 00:31:02.801796102 +0000 UTC m=+3727.901396263"
Nov 04 00:31:02.821078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.821044    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 04 00:31:02.857380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.857348    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 04 00:31:02.949856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.949816    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:02.965424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965392    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.965615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965432    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.965615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965466    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.965615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965491    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.965615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965529    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.965615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965549    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.965615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965587    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965676    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965710    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965812    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965839    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965884    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965902    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965937    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965969    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966008    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.966023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.965950    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:02.966600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966067    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966067    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966246    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966255    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966290    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token\") pod \"951a1a46-d504-4b34-9f6b-12d2a57ba342\" (UID: \"951a1a46-d504-4b34-9f6b-12d2a57ba342\") "
Nov 04 00:31:02.966600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966356    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket\") pod \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\" (UID: \"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33\") "
Nov 04 00:31:02.966984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966292    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966894    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966410    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966941    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.966984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.966600    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967103    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967124    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967141    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967164    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967180    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967193    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967206    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967220    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967234    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a1a46-d504-4b34-9f6b-12d2a57ba342-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.967268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.967248    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:02.968695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.968667    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "kube-api-access-p84g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:02.969228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.969190    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:31:02.969459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.969428    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.969886 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.969858    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:02.970177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.970152    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "kube-api-access-58pcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:02.970279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.970263    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:31:02.970552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.970534    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token" pod "951a1a46-d504-4b34-9f6b-12d2a57ba342" (UID: "951a1a46-d504-4b34-9f6b-12d2a57ba342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:02.970653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:02.970635    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token" pod "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" (UID: "d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:03.068902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068866    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84g2\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-kube-api-access-p84g2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.068902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068900    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.069135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068911    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.069135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068920    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.069135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068929    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.069135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068938    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a1a46-d504-4b34-9f6b-12d2a57ba342-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.069135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068947    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.069135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.068956    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58pcd\" (UniqueName: \"kubernetes.io/projected/951a1a46-d504-4b34-9f6b-12d2a57ba342-kube-api-access-58pcd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:03.779264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.779225    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"
Nov 04 00:31:03.779264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.779253    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4" event={"ID":"d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33","Type":"ContainerDied","Data":"208200bc0ab1941358932465fb2c8c21410c1562566fcbf0ef69c93264fe032f"}
Nov 04 00:31:03.779860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.779315    5074 scope.go:122] "RemoveContainer" containerID="db72f39c672b78c740140110f7129a0589f2e4365e05973dc587dc2caf90c872"
Nov 04 00:31:03.781060 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.781036    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"
Nov 04 00:31:03.781169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.781057    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq" event={"ID":"951a1a46-d504-4b34-9f6b-12d2a57ba342","Type":"ContainerDied","Data":"ed0be28b0ec1090c8e5c4c64710683586b0f7d1387974a9ac8bee3324e86e582"}
Nov 04 00:31:03.792538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.792511    5074 scope.go:122] "RemoveContainer" containerID="0fabca77f932d902cb63f9adbc9c1c5f73a51001be06db8477f3467ecf4a9601"
Nov 04 00:31:03.806288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.806264    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"]
Nov 04 00:31:03.812319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.812288    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-6lktq"]
Nov 04 00:31:03.825325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.825297    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"]
Nov 04 00:31:03.831138 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:03.831115    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-ptdl4"]
Nov 04 00:31:04.028757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:04.028704    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"]
Nov 04 00:31:04.032797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:04.032707    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927315ed-qrvhs"]
Nov 04 00:31:05.020648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:05.020612    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65935fe5-8739-4fe3-b88b-567944039c03" path="/var/lib/kubelet/pods/65935fe5-8739-4fe3-b88b-567944039c03/volumes"
Nov 04 00:31:05.021200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:05.021183    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" path="/var/lib/kubelet/pods/951a1a46-d504-4b34-9f6b-12d2a57ba342/volumes"
Nov 04 00:31:05.021653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:05.021639    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" path="/var/lib/kubelet/pods/d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33/volumes"
Nov 04 00:31:09.788396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:09.788347    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" podUID="e90267b5-38ea-499b-aa99-2eb911054653" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:31:10.012784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:10.011650    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"]
Nov 04 00:31:10.021857 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:10.021821    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"]
Nov 04 00:31:11.949891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:11.949856    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:11.962849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:11.962822    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:12.872253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:12.872221    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:31:13.587136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.587101    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"]
Nov 04 00:31:13.587891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.587873    5074 cpu_manager.go:414] "containerMap: removing container" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy"
Nov 04 00:31:13.587891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.587891    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy"
Nov 04 00:31:13.588064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.587907    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy"
Nov 04 00:31:13.588064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.587913    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy"
Nov 04 00:31:13.588064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.588053    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="951a1a46-d504-4b34-9f6b-12d2a57ba342" containerName="istio-proxy"
Nov 04 00:31:13.588064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.588063    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cc8da7-16d5-41d0-93b7-b34ad0dc0f33" containerName="istio-proxy"
Nov 04 00:31:13.593500 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.593475    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.606605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.606572    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"]
Nov 04 00:31:13.609061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.609021    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"openshift-service-ca.crt\""
Nov 04 00:31:13.609327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.609308    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"rate-test-model-4331a7-kserve-self-signed-certs\""
Nov 04 00:31:13.609460 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.609390    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"default-dockercfg-qz7cn\""
Nov 04 00:31:13.609531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.609486    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4331a7\"/\"kube-root-ca.crt\""
Nov 04 00:31:13.734976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.734938    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.735174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.735053    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zb2\" (UniqueName: \"kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.735174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.735115    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.735307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.735186    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.735307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.735290    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.735417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.735375    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.836776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.836720    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.837005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.836865    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.837005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.836943    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zb2\" (UniqueName: \"kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.837125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.837000    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.837125 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.837090    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.837266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.837211    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.837732 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.837710    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.838031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.838010    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.839062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.839034    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.843141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.843116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.844293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.844269    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.852882 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.852838    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zb2\" (UniqueName: \"kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2\") pod \"rate-test-model-4331a7-kserve-6f79949649-wpsxr\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") " pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:13.928379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:13.928316    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:14.057888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.057788    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"]
Nov 04 00:31:14.064119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.064076    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c0a7cf-5sgwd"]
Nov 04 00:31:14.244721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.244692    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"]
Nov 04 00:31:14.244913 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:31:14.244833    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0a9039b_60d1_484d_b01e_716f1def4cf7.slice/crio-b29907971b96ebba98d142adcdff73608ceee7ced227ca61ff88b5a6a765f167 WatchSource:0}: Error finding container b29907971b96ebba98d142adcdff73608ceee7ced227ca61ff88b5a6a765f167: Status 404 returned error can't find the container with id b29907971b96ebba98d142adcdff73608ceee7ced227ca61ff88b5a6a765f167
Nov 04 00:31:14.884069 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.884034    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr" event={"ID":"f0a9039b-60d1-484d-b01e-716f1def4cf7","Type":"ContainerStarted","Data":"e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b"}
Nov 04 00:31:14.884482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.884077    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr" event={"ID":"f0a9039b-60d1-484d-b01e-716f1def4cf7","Type":"ContainerStarted","Data":"b29907971b96ebba98d142adcdff73608ceee7ced227ca61ff88b5a6a765f167"}
Nov 04 00:31:14.906550 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.906491    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr" podStartSLOduration=1.906474024 podStartE2EDuration="1.906474024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:31:14.903486717 +0000 UTC m=+3740.003086861" watchObservedRunningTime="2026-11-04 00:31:14.906474024 +0000 UTC m=+3740.006074170"
Nov 04 00:31:14.929225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:14.929189    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:15.012393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.012337    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s" podUID="5ed92c25-9518-438e-b30a-175d7aa94e3c" containerName="payload-processing" containerID="cri-o://e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767" gracePeriod=25
Nov 04 00:31:15.034642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.024978    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerName="payload-pre-processing" containerID="cri-o://71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920" gracePeriod=25
Nov 04 00:31:15.043493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.043456    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbe28bb-5abb-42f2-b330-5a05cf5e4375" path="/var/lib/kubelet/pods/dcbe28bb-5abb-42f2-b330-5a05cf5e4375/volumes"
Nov 04 00:31:15.329510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.329481    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:31:15.470068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.470035    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs\" (UniqueName: \"kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs\") pod \"5ed92c25-9518-438e-b30a-175d7aa94e3c\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") "
Nov 04 00:31:15.470287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.470234    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume\") pod \"5ed92c25-9518-438e-b30a-175d7aa94e3c\" (UID: \"5ed92c25-9518-438e-b30a-175d7aa94e3c\") "
Nov 04 00:31:15.470605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.470570    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume" pod "5ed92c25-9518-438e-b30a-175d7aa94e3c" (UID: "5ed92c25-9518-438e-b30a-175d7aa94e3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:15.470717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.470693    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed92c25-9518-438e-b30a-175d7aa94e3c-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:15.472219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.472199    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs" pod "5ed92c25-9518-438e-b30a-175d7aa94e3c" (UID: "5ed92c25-9518-438e-b30a-175d7aa94e3c"). InnerVolumeSpecName "kube-api-access-5h2hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:15.572538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.572494    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h2hs\" (UniqueName: \"kubernetes.io/projected/5ed92c25-9518-438e-b30a-175d7aa94e3c-kube-api-access-5h2hs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:15.893533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.893442    5074 generic.go:356] "Generic (PLEG): container finished" podID="5ed92c25-9518-438e-b30a-175d7aa94e3c" containerID="e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767" exitCode=0
Nov 04 00:31:15.893533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.893491    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"
Nov 04 00:31:15.893533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.893521    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s" event={"ID":"5ed92c25-9518-438e-b30a-175d7aa94e3c","Type":"ContainerDied","Data":"e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767"}
Nov 04 00:31:15.894073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.893577    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s" event={"ID":"5ed92c25-9518-438e-b30a-175d7aa94e3c","Type":"ContainerDied","Data":"e72d230deaaf23c84ea60faa6f178d759a08dbc10e4a94129409a2b1bd7d91cc"}
Nov 04 00:31:15.894073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.893604    5074 scope.go:122] "RemoveContainer" containerID="e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767"
Nov 04 00:31:15.905528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.905508    5074 scope.go:122] "RemoveContainer" containerID="e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767"
Nov 04 00:31:15.905845 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:31:15.905819    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767\": container with ID starting with e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767 not found: ID does not exist" containerID="e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767"
Nov 04 00:31:15.905932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.905853    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767"} err="failed to get container status \"e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767\": rpc error: code = NotFound desc = could not find container \"e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767\": container with ID starting with e895e041f1e0a316f7296ecf9fd9ca8beaa21970c5a0c84b3e87e4c5bd566767 not found: ID does not exist"
Nov 04 00:31:15.922190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.922159    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"]
Nov 04 00:31:15.925917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:15.925890    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5fca-57f6d5df66-zc52s"]
Nov 04 00:31:16.700299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.700269    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:31:16.789499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.789466    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume\") pod \"e90267b5-38ea-499b-aa99-2eb911054653\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") "
Nov 04 00:31:16.789657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.789626    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h\" (UniqueName: \"kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h\") pod \"e90267b5-38ea-499b-aa99-2eb911054653\" (UID: \"e90267b5-38ea-499b-aa99-2eb911054653\") "
Nov 04 00:31:16.789904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.789880    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume" pod "e90267b5-38ea-499b-aa99-2eb911054653" (UID: "e90267b5-38ea-499b-aa99-2eb911054653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:16.790079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.790063    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90267b5-38ea-499b-aa99-2eb911054653-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:16.791796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.791766    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h" pod "e90267b5-38ea-499b-aa99-2eb911054653" (UID: "e90267b5-38ea-499b-aa99-2eb911054653"). InnerVolumeSpecName "kube-api-access-l2b4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:16.891698 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.891658    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2b4h\" (UniqueName: \"kubernetes.io/projected/e90267b5-38ea-499b-aa99-2eb911054653-kube-api-access-l2b4h\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:16.903002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.902968    5074 generic.go:356] "Generic (PLEG): container finished" podID="e90267b5-38ea-499b-aa99-2eb911054653" containerID="7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460" exitCode=0
Nov 04 00:31:16.903473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.903014    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"
Nov 04 00:31:16.903473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.903038    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" event={"ID":"e90267b5-38ea-499b-aa99-2eb911054653","Type":"ContainerDied","Data":"7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460"}
Nov 04 00:31:16.903473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.903092    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5" event={"ID":"e90267b5-38ea-499b-aa99-2eb911054653","Type":"ContainerDied","Data":"57fe59cb41d4edae251f59474c534c07522f8976393389765794e909a4c357d6"}
Nov 04 00:31:16.903473 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.903122    5074 scope.go:122] "RemoveContainer" containerID="7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460"
Nov 04 00:31:16.917247 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.917197    5074 scope.go:122] "RemoveContainer" containerID="7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460"
Nov 04 00:31:16.917531 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:31:16.917507    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460\": container with ID starting with 7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460 not found: ID does not exist" containerID="7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460"
Nov 04 00:31:16.917633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.917542    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460"} err="failed to get container status \"7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460\": rpc error: code = NotFound desc = could not find container \"7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460\": container with ID starting with 7f04ac82a7947b1d716f98946c98462c08bd8360bbd1e56175b7bdca72ad1460 not found: ID does not exist"
Nov 04 00:31:16.932262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.932195    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"]
Nov 04 00:31:16.936450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:16.936422    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a002b-5fd4dc8d47-jv7d5"]
Nov 04 00:31:17.022074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:17.022037    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed92c25-9518-438e-b30a-175d7aa94e3c" path="/var/lib/kubelet/pods/5ed92c25-9518-438e-b30a-175d7aa94e3c/volumes"
Nov 04 00:31:17.022689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:17.022670    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90267b5-38ea-499b-aa99-2eb911054653" path="/var/lib/kubelet/pods/e90267b5-38ea-499b-aa99-2eb911054653/volumes"
Nov 04 00:31:22.034021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:22.033971    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:31:23.929165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:23.929127    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:23.946368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:23.946336    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:24.018699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:24.012445    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:31:27.416842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.416789    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"]
Nov 04 00:31:27.417975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.417939    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e90267b5-38ea-499b-aa99-2eb911054653" containerName="payload-pre-processing"
Nov 04 00:31:27.417975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.417968    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90267b5-38ea-499b-aa99-2eb911054653" containerName="payload-pre-processing"
Nov 04 00:31:27.418217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.418039    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5ed92c25-9518-438e-b30a-175d7aa94e3c" containerName="payload-processing"
Nov 04 00:31:27.418217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.418048    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed92c25-9518-438e-b30a-175d7aa94e3c" containerName="payload-processing"
Nov 04 00:31:27.418217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.418212    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed92c25-9518-438e-b30a-175d7aa94e3c" containerName="payload-processing"
Nov 04 00:31:27.418381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.418236    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90267b5-38ea-499b-aa99-2eb911054653" containerName="payload-pre-processing"
Nov 04 00:31:27.424428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.424399    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.431102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.431077    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-58b75d-openshift-default-dockercfg-ks9wt\""
Nov 04 00:31:27.435428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.435394    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"]
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442224    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442289    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbtr\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442345    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442384    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442425    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442454    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442580    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442641    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.449795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.442675    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.543587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.543542    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.543587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.543587    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.543892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.543708    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.543892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.543770    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.543905    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.543980    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544023    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544148 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544140    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544183    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544192    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbtr\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544261    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544022    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544371    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.544524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.544433    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.546543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.546518    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.546888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.546867    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.556805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.556781    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.556922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.556905    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbtr\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:27.760465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:27.760435    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:28.025956 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:31:28.025857    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:31:28.060246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:28.060205    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"]
Nov 04 00:31:28.062064 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:31:28.062035    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aa3221_e5de_45b7_b53d_329c239d6b7f.slice/crio-0202ad6245cf5f81b58e402de5ef1e004851d0e52b8d07d24fd7194d63d8b230 WatchSource:0}: Error finding container 0202ad6245cf5f81b58e402de5ef1e004851d0e52b8d07d24fd7194d63d8b230: Status 404 returned error can't find the container with id 0202ad6245cf5f81b58e402de5ef1e004851d0e52b8d07d24fd7194d63d8b230
Nov 04 00:31:28.065678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:28.065637    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:31:28.065813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:28.065728    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:31:28.065813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:28.065787    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:31:29.044190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:29.044145    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" event={"ID":"67aa3221-e5de-45b7-b53d-329c239d6b7f","Type":"ContainerStarted","Data":"d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2"}
Nov 04 00:31:29.044646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:29.044218    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" event={"ID":"67aa3221-e5de-45b7-b53d-329c239d6b7f","Type":"ContainerStarted","Data":"0202ad6245cf5f81b58e402de5ef1e004851d0e52b8d07d24fd7194d63d8b230"}
Nov 04 00:31:29.072863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:29.072805    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podStartSLOduration=2.072789389 podStartE2EDuration="2.072789389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:31:29.070476585 +0000 UTC m=+3754.170076730" watchObservedRunningTime="2026-11-04 00:31:29.072789389 +0000 UTC m=+3754.172389535"
Nov 04 00:31:29.760730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:29.760689    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:29.762866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:29.762840    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused" start-of-body=
Nov 04 00:31:29.762955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:29.762897    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused"
Nov 04 00:31:30.188703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.188657    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"]
Nov 04 00:31:30.195393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.195357    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.225758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.216446    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"default-dockercfg-gwkft\""
Nov 04 00:31:30.225758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.218987    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"openshift-service-ca.crt\""
Nov 04 00:31:30.225758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.219163    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"rate-test-model-398158-kserve-self-signed-certs\""
Nov 04 00:31:30.225758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.219819    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-398158\"/\"kube-root-ca.crt\""
Nov 04 00:31:30.226997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.226970    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"]
Nov 04 00:31:30.297012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.296974    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.297228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.297112    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.297308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.297252    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.297404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.297374    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.297477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.297419    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdh52\" (UniqueName: \"kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.297539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.297476    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.405950    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.406024    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdh52\" (UniqueName: \"kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.406077    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.406124    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.406206    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.406313    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.407760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.407269    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.408438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.408151    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.408438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.408376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.417625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.417045    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.417830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.417628    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.427725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.427697    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdh52\" (UniqueName: \"kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52\") pod \"rate-test-model-398158-kserve-6d58775ff7-52s84\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") " pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.569377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.569293    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:30.767527 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.767492    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused" start-of-body=
Nov 04 00:31:30.767867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.767831    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused"
Nov 04 00:31:30.969724 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:30.969693    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"]
Nov 04 00:31:30.970172 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:31:30.970143    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22c2d6be_9efd_4f5d_80a1_b5b196a57b8a.slice/crio-db76d7f90bf3ef9d6b05361a9970fe604d3572925fcda61519d8e572a422ea6e WatchSource:0}: Error finding container db76d7f90bf3ef9d6b05361a9970fe604d3572925fcda61519d8e572a422ea6e: Status 404 returned error can't find the container with id db76d7f90bf3ef9d6b05361a9970fe604d3572925fcda61519d8e572a422ea6e
Nov 04 00:31:31.069671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:31.069632    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" event={"ID":"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a","Type":"ContainerStarted","Data":"db76d7f90bf3ef9d6b05361a9970fe604d3572925fcda61519d8e572a422ea6e"}
Nov 04 00:31:31.761462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:31.761407    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused" start-of-body=
Nov 04 00:31:31.761920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:31.761830    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused"
Nov 04 00:31:32.035327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:32.035236    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:31:32.086351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:32.086312    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" event={"ID":"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a","Type":"ContainerStarted","Data":"842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9"}
Nov 04 00:31:32.146765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:32.146496    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" podStartSLOduration=2.146473003 podStartE2EDuration="2.146473003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:31:32.108170183 +0000 UTC m=+3757.207770332" watchObservedRunningTime="2026-11-04 00:31:32.146473003 +0000 UTC m=+3757.246073148"
Nov 04 00:31:32.569679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:32.569642    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:32.761648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:32.761614    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused" start-of-body=
Nov 04 00:31:32.762080 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:32.761664    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused"
Nov 04 00:31:33.761633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:33.761578    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused" start-of-body=
Nov 04 00:31:33.761833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:33.761683    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.164:15021/healthz/ready\": dial tcp 10.132.1.164:15021: connect: connection refused"
Nov 04 00:31:34.767412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:34.767380    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:34.767925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:34.767897    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:34.768591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:34.768571    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:31:35.348057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.348027    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:31:35.412480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.412445    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42\" (UniqueName: \"kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42\") pod \"430961a4-6dcd-4bff-be54-addc4e3cade4\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") "
Nov 04 00:31:35.412671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.412570    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume\") pod \"430961a4-6dcd-4bff-be54-addc4e3cade4\" (UID: \"430961a4-6dcd-4bff-be54-addc4e3cade4\") "
Nov 04 00:31:35.413117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.413088    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume" pod "430961a4-6dcd-4bff-be54-addc4e3cade4" (UID: "430961a4-6dcd-4bff-be54-addc4e3cade4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:35.413271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.413251    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430961a4-6dcd-4bff-be54-addc4e3cade4-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:35.414890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.414865    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42" pod "430961a4-6dcd-4bff-be54-addc4e3cade4" (UID: "430961a4-6dcd-4bff-be54-addc4e3cade4"). InnerVolumeSpecName "kube-api-access-47c42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:35.514580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:35.514545    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47c42\" (UniqueName: \"kubernetes.io/projected/430961a4-6dcd-4bff-be54-addc4e3cade4-kube-api-access-47c42\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:36.126623 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.126582    5074 generic.go:356] "Generic (PLEG): container finished" podID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerID="71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920" exitCode=0
Nov 04 00:31:36.127099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.126642    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"
Nov 04 00:31:36.127099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.126669    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" event={"ID":"430961a4-6dcd-4bff-be54-addc4e3cade4","Type":"ContainerDied","Data":"71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920"}
Nov 04 00:31:36.127099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.126730    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t" event={"ID":"430961a4-6dcd-4bff-be54-addc4e3cade4","Type":"ContainerDied","Data":"561ad9b866a8dcedc5e67998aa49df9e37ac014d00a83d00b3dd281a0332ea9c"}
Nov 04 00:31:36.127099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.126766    5074 scope.go:122] "RemoveContainer" containerID="71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920"
Nov 04 00:31:36.139727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.139702    5074 scope.go:122] "RemoveContainer" containerID="71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920"
Nov 04 00:31:36.140083 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:31:36.140048    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920\": container with ID starting with 71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920 not found: ID does not exist" containerID="71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920"
Nov 04 00:31:36.140169 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.140095    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920"} err="failed to get container status \"71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920\": rpc error: code = NotFound desc = could not find container \"71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920\": container with ID starting with 71b4a668ce20cbb2336e1f0b1efa6cbe69ccbe47807951f8bd6a855f156d5920 not found: ID does not exist"
Nov 04 00:31:36.156164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.156127    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"]
Nov 04 00:31:36.159830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:36.159805    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5fca-7bf599cbd7-bnk6t"]
Nov 04 00:31:37.021858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:37.021824    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" path="/var/lib/kubelet/pods/430961a4-6dcd-4bff-be54-addc4e3cade4/volumes"
Nov 04 00:31:40.570676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:40.570638    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:40.584601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:40.584572    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:41.181262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:41.181225    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:31:57.039104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:57.039058    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"]
Nov 04 00:31:57.043433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:57.043405    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199e093-kd5qj"]
Nov 04 00:31:58.032779 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:31:58.032706    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:31:59.022241 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:59.022205    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d091ca77-9bad-4ace-860c-95fb01d4b06c" path="/var/lib/kubelet/pods/d091ca77-9bad-4ace-860c-95fb01d4b06c/volumes"
Nov 04 00:31:59.237812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:59.237779    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"]
Nov 04 00:31:59.251082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:31:59.251045    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"]
Nov 04 00:32:01.226215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:01.226180    5074 scope.go:122] "RemoveContainer" containerID="8436bcc8be5f82f0e9b569b3d005bee71a163c1f41fba471f01ea19b953d9bc7"
Nov 04 00:32:04.239094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.239052    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" podUID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" containerName="payload-processing" containerID="cri-o://ad2076d4450aee90eae70f5f0d2578f886b26615c5ba846b26bc5905349cfd01" gracePeriod=25
Nov 04 00:32:04.251674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.251638    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" podUID="6fea0052-1521-4874-82f8-bcf7b425172f" containerName="payload-pre-processing" containerID="cri-o://32ab51e9f2d163ff4256c5757e492279ee171674a97780bd6406b6802d45a2ab" gracePeriod=25
Nov 04 00:32:04.283280 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:32:04.283259    5074 logging.go:55] [core] [Channel #24381 SubChannel #24382]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.179:50052", ServerName: "10.132.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.179:50052: connect: connection refused"
Nov 04 00:32:04.297251 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:32:04.297229    5074 logging.go:55] [core] [Channel #24383 SubChannel #24384]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: connect: connection refused"
Nov 04 00:32:04.392806 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.392772    5074 generic.go:356] "Generic (PLEG): container finished" podID="6fea0052-1521-4874-82f8-bcf7b425172f" containerID="32ab51e9f2d163ff4256c5757e492279ee171674a97780bd6406b6802d45a2ab" exitCode=0
Nov 04 00:32:04.393022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.392862    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" event={"ID":"6fea0052-1521-4874-82f8-bcf7b425172f","Type":"ContainerDied","Data":"32ab51e9f2d163ff4256c5757e492279ee171674a97780bd6406b6802d45a2ab"}
Nov 04 00:32:04.399775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.398575    5074 generic.go:356] "Generic (PLEG): container finished" podID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" containerID="ad2076d4450aee90eae70f5f0d2578f886b26615c5ba846b26bc5905349cfd01" exitCode=0
Nov 04 00:32:04.399775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.398907    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" event={"ID":"e546e4c8-8ed6-41d2-b44d-ede826ff75d1","Type":"ContainerDied","Data":"ad2076d4450aee90eae70f5f0d2578f886b26615c5ba846b26bc5905349cfd01"}
Nov 04 00:32:04.579501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.579469    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 04 00:32:04.658173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.658138    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 04 00:32:04.769678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.769630    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume\") pod \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") "
Nov 04 00:32:04.769899 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.769720    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume\") pod \"6fea0052-1521-4874-82f8-bcf7b425172f\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") "
Nov 04 00:32:04.769899 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.769794    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g\" (UniqueName: \"kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g\") pod \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\" (UID: \"e546e4c8-8ed6-41d2-b44d-ede826ff75d1\") "
Nov 04 00:32:04.770059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.770034    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj\" (UniqueName: \"kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj\") pod \"6fea0052-1521-4874-82f8-bcf7b425172f\" (UID: \"6fea0052-1521-4874-82f8-bcf7b425172f\") "
Nov 04 00:32:04.770310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.770286    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume" pod "e546e4c8-8ed6-41d2-b44d-ede826ff75d1" (UID: "e546e4c8-8ed6-41d2-b44d-ede826ff75d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:04.770446 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.770288    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume" pod "6fea0052-1521-4874-82f8-bcf7b425172f" (UID: "6fea0052-1521-4874-82f8-bcf7b425172f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:04.770686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.770660    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:04.770686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.770685    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea0052-1521-4874-82f8-bcf7b425172f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:04.772182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.772154    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g" pod "e546e4c8-8ed6-41d2-b44d-ede826ff75d1" (UID: "e546e4c8-8ed6-41d2-b44d-ede826ff75d1"). InnerVolumeSpecName "kube-api-access-qpb2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:04.772269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.772194    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj" pod "6fea0052-1521-4874-82f8-bcf7b425172f" (UID: "6fea0052-1521-4874-82f8-bcf7b425172f"). InnerVolumeSpecName "kube-api-access-lnhpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:04.873085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.873005    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpb2g\" (UniqueName: \"kubernetes.io/projected/e546e4c8-8ed6-41d2-b44d-ede826ff75d1-kube-api-access-qpb2g\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:04.873085 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:04.873038    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhpj\" (UniqueName: \"kubernetes.io/projected/6fea0052-1521-4874-82f8-bcf7b425172f-kube-api-access-lnhpj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:05.283059 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.283016    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" podUID="6fea0052-1521-4874-82f8-bcf7b425172f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.179:50052\" within 1s: context deadline exceeded"
Nov 04 00:32:05.297689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.297656    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" podUID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.178:50052\" within 1s: context deadline exceeded"
Nov 04 00:32:05.297845 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:32:05.297757    5074 logging.go:55] [core] [Channel #24383 SubChannel #24384]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: operation was canceled"
Nov 04 00:32:05.413493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.413458    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"
Nov 04 00:32:05.413731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.413461    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q" event={"ID":"6fea0052-1521-4874-82f8-bcf7b425172f","Type":"ContainerDied","Data":"58c1f61f9fa96f77071c9140db54bbd3e87c6cfa3ae6aa0569172cebb6a566cd"}
Nov 04 00:32:05.413977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.413948    5074 scope.go:122] "RemoveContainer" containerID="32ab51e9f2d163ff4256c5757e492279ee171674a97780bd6406b6802d45a2ab"
Nov 04 00:32:05.418176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.417919    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx" event={"ID":"e546e4c8-8ed6-41d2-b44d-ede826ff75d1","Type":"ContainerDied","Data":"436c6a52b20cd540f68acace2c3b4ee9c01ca9d4fd4f6e4cf7e1c55658d70e52"}
Nov 04 00:32:05.418176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.417988    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"
Nov 04 00:32:05.428707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.428685    5074 scope.go:122] "RemoveContainer" containerID="ad2076d4450aee90eae70f5f0d2578f886b26615c5ba846b26bc5905349cfd01"
Nov 04 00:32:05.457879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.457855    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"]
Nov 04 00:32:05.461606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.461582    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a5948-55ddb68ff4-w8lcx"]
Nov 04 00:32:05.473404 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.473384    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"]
Nov 04 00:32:05.480632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:05.480608    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a5948-8959bd55f-sqx7q"]
Nov 04 00:32:07.019576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:07.019537    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fea0052-1521-4874-82f8-bcf7b425172f" path="/var/lib/kubelet/pods/6fea0052-1521-4874-82f8-bcf7b425172f/volumes"
Nov 04 00:32:07.020344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:07.020318    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" path="/var/lib/kubelet/pods/e546e4c8-8ed6-41d2-b44d-ede826ff75d1/volumes"
Nov 04 00:32:10.393276 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:10.393239    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"]
Nov 04 00:32:10.406717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:10.406687    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"]
Nov 04 00:32:15.393560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.393519    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht" podUID="b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" containerName="payload-processing" containerID="cri-o://7c3a41c98d2a04232112d7371289f2a68cb75d008c82bbbb6f28e44d97f4f77a" gracePeriod=25
Nov 04 00:32:15.407846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.407781    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" containerName="payload-pre-processing" containerID="cri-o://f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903" gracePeriod=25
Nov 04 00:32:15.531798 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.531761    5074 generic.go:356] "Generic (PLEG): container finished" podID="b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" containerID="7c3a41c98d2a04232112d7371289f2a68cb75d008c82bbbb6f28e44d97f4f77a" exitCode=0
Nov 04 00:32:15.531964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.531942    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht" event={"ID":"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc","Type":"ContainerDied","Data":"7c3a41c98d2a04232112d7371289f2a68cb75d008c82bbbb6f28e44d97f4f77a"}
Nov 04 00:32:15.705982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.705945    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:32:15.840612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.840572    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume\") pod \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") "
Nov 04 00:32:15.840935 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.840891    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr\" (UniqueName: \"kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr\") pod \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\" (UID: \"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc\") "
Nov 04 00:32:15.842312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.842269    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume" pod "b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" (UID: "b160dbb1-fdd5-4d63-add3-03fb9bdebbcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:15.844438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.844410    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr" pod "b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" (UID: "b160dbb1-fdd5-4d63-add3-03fb9bdebbcc"). InnerVolumeSpecName "kube-api-access-tjcpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:15.951197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.946777    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:15.951197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:15.946818    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcpr\" (UniqueName: \"kubernetes.io/projected/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc-kube-api-access-tjcpr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:16.542963 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:16.542918    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht" event={"ID":"b160dbb1-fdd5-4d63-add3-03fb9bdebbcc","Type":"ContainerDied","Data":"7760df4e305ad3b67a3215b0874ed38563ffe52b5f3127f34669f7660f3d6b8a"}
Nov 04 00:32:16.543507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:16.543020    5074 scope.go:122] "RemoveContainer" containerID="7c3a41c98d2a04232112d7371289f2a68cb75d008c82bbbb6f28e44d97f4f77a"
Nov 04 00:32:16.543507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:16.542934    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"
Nov 04 00:32:16.572464 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:16.572413    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"]
Nov 04 00:32:16.576374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:16.576336    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7d218-6c687f5696-tx8ht"]
Nov 04 00:32:17.025438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:17.025398    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" path="/var/lib/kubelet/pods/b160dbb1-fdd5-4d63-add3-03fb9bdebbcc/volumes"
Nov 04 00:32:19.432319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.432233    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"]
Nov 04 00:32:19.433421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433392    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" containerName="payload-processing"
Nov 04 00:32:19.433421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433421    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" containerName="payload-processing"
Nov 04 00:32:19.433603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433465    5074 cpu_manager.go:414] "containerMap: removing container" podUID="6fea0052-1521-4874-82f8-bcf7b425172f" containerName="payload-pre-processing"
Nov 04 00:32:19.433603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433474    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fea0052-1521-4874-82f8-bcf7b425172f" containerName="payload-pre-processing"
Nov 04 00:32:19.433603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433489    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" containerName="payload-processing"
Nov 04 00:32:19.433603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433497    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" containerName="payload-processing"
Nov 04 00:32:19.433603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433520    5074 cpu_manager.go:414] "containerMap: removing container" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerName="payload-pre-processing"
Nov 04 00:32:19.433603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433528    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerName="payload-pre-processing"
Nov 04 00:32:19.433955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433700    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fea0052-1521-4874-82f8-bcf7b425172f" containerName="payload-pre-processing"
Nov 04 00:32:19.433955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433715    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b160dbb1-fdd5-4d63-add3-03fb9bdebbcc" containerName="payload-processing"
Nov 04 00:32:19.433955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433728    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="430961a4-6dcd-4bff-be54-addc4e3cade4" containerName="payload-pre-processing"
Nov 04 00:32:19.433955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.433765    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e546e4c8-8ed6-41d2-b44d-ede826ff75d1" containerName="payload-processing"
Nov 04 00:32:19.454907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.441440    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.454907 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.454501    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"]
Nov 04 00:32:19.458007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.457980    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:32:19.458253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.458239    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:32:19.610884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.610840    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.611141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.610962    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbsj\" (UniqueName: \"kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.611141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.611075    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.712659 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.712619    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.712897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.712678    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbsj\" (UniqueName: \"kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.712897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.712787    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.713414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.713364    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.715332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.715308    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.729162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.729136    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbsj\" (UniqueName: \"kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj\") pod \"maas-api-revoke-sub-9fac65e7-9z6kb\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:19.767448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:19.767423    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:20.063465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:20.063434    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"]
Nov 04 00:32:20.064206 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:32:20.064180    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fe2897_75b6_4c46_a5b6_db49e6969dc4.slice/crio-621cff7e985a9468383b7a706538ad75394e98aad2271a88658e2522ff29c078 WatchSource:0}: Error finding container 621cff7e985a9468383b7a706538ad75394e98aad2271a88658e2522ff29c078: Status 404 returned error can't find the container with id 621cff7e985a9468383b7a706538ad75394e98aad2271a88658e2522ff29c078
Nov 04 00:32:20.596231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:20.596142    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb" event={"ID":"c6fe2897-75b6-4c46-a5b6-db49e6969dc4","Type":"ContainerStarted","Data":"621cff7e985a9468383b7a706538ad75394e98aad2271a88658e2522ff29c078"}
Nov 04 00:32:21.607811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:21.607704    5074 generic.go:356] "Generic (PLEG): container finished" podID="c6fe2897-75b6-4c46-a5b6-db49e6969dc4" containerID="6a4750097337aed7fd7b8b929b62c2a0d40cef3a479b0e87651a59264d69b45d" exitCode=0
Nov 04 00:32:21.608523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:21.607988    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb" event={"ID":"c6fe2897-75b6-4c46-a5b6-db49e6969dc4","Type":"ContainerDied","Data":"6a4750097337aed7fd7b8b929b62c2a0d40cef3a479b0e87651a59264d69b45d"}
Nov 04 00:32:22.029528 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.029488    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"]
Nov 04 00:32:22.033183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.033156    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20b8b04-mmlcj"]
Nov 04 00:32:22.798355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.798327    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:22.965412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.965376    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj\" (UniqueName: \"kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj\") pod \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") "
Nov 04 00:32:22.965593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.965505    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca\") pod \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") "
Nov 04 00:32:22.965593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.965573    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca\") pod \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\" (UID: \"c6fe2897-75b6-4c46-a5b6-db49e6969dc4\") "
Nov 04 00:32:22.965908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.965883    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca" pod "c6fe2897-75b6-4c46-a5b6-db49e6969dc4" (UID: "c6fe2897-75b6-4c46-a5b6-db49e6969dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:22.966131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.966100    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:22.967755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.967714    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca" pod "c6fe2897-75b6-4c46-a5b6-db49e6969dc4" (UID: "c6fe2897-75b6-4c46-a5b6-db49e6969dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:22.967850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:22.967725    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj" pod "c6fe2897-75b6-4c46-a5b6-db49e6969dc4" (UID: "c6fe2897-75b6-4c46-a5b6-db49e6969dc4"). InnerVolumeSpecName "kube-api-access-nmbsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:23.022956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.022917    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e5567f-f8b8-427d-8592-66c3c90a8c2c" path="/var/lib/kubelet/pods/71e5567f-f8b8-427d-8592-66c3c90a8c2c/volumes"
Nov 04 00:32:23.067265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.067231    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.067265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.067263    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbsj\" (UniqueName: \"kubernetes.io/projected/c6fe2897-75b6-4c46-a5b6-db49e6969dc4-kube-api-access-nmbsj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.630412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.630374    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb" event={"ID":"c6fe2897-75b6-4c46-a5b6-db49e6969dc4","Type":"ContainerDied","Data":"621cff7e985a9468383b7a706538ad75394e98aad2271a88658e2522ff29c078"}
Nov 04 00:32:23.630683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.630429    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621cff7e985a9468383b7a706538ad75394e98aad2271a88658e2522ff29c078"
Nov 04 00:32:23.630683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.630481    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"
Nov 04 00:32:23.985196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:23.985150    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:32:24.079579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:24.079544    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"]
Nov 04 00:32:26.233309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:26.233267    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"]
Nov 04 00:32:26.239505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:26.239470    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"]
Nov 04 00:32:28.024794 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:28.024724    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:32:30.235509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.235465    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"]
Nov 04 00:32:30.237956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.236364    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c6fe2897-75b6-4c46-a5b6-db49e6969dc4" containerName="revoke-keys"
Nov 04 00:32:30.237956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.236384    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fe2897-75b6-4c46-a5b6-db49e6969dc4" containerName="revoke-keys"
Nov 04 00:32:30.237956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.236511    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fe2897-75b6-4c46-a5b6-db49e6969dc4" containerName="revoke-keys"
Nov 04 00:32:30.240209 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.240189    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.244775 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.244711    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:32:30.244897 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.244846    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:32:30.248009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.247988    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"]
Nov 04 00:32:30.288559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.288518    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.288786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.288731    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.288863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.288839    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.391422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.391364    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.391626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.391456    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.391689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.391641    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.392259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.392237    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.394458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.394432    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.406790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.406693    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99\") pod \"maas-api-revoke-sub-5cfa9eba-hsb7p\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.565678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.565589    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:30.852990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:30.852955    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"]
Nov 04 00:32:30.853671 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:32:30.853633    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5385bcd5_a251_4445_9d38_b99d94c1b17e.slice/crio-34b4223b2ce236a26ce81e97ba8c94e62a59a5d2fd2dfa20ea00d24641586b77 WatchSource:0}: Error finding container 34b4223b2ce236a26ce81e97ba8c94e62a59a5d2fd2dfa20ea00d24641586b77: Status 404 returned error can't find the container with id 34b4223b2ce236a26ce81e97ba8c94e62a59a5d2fd2dfa20ea00d24641586b77
Nov 04 00:32:31.234171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.234129    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t" podUID="737b7b5e-1188-4b76-9cea-fc60e57d4165" containerName="payload-processing" containerID="cri-o://2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f" gracePeriod=25
Nov 04 00:32:31.240677 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.240634    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb" podUID="295a683e-40ef-4e48-85b1-25f3d1603f52" containerName="payload-pre-processing" containerID="cri-o://6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5" gracePeriod=25
Nov 04 00:32:31.635009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.634970    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 04 00:32:31.638585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.638562    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 04 00:32:31.715799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.708996    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz\" (UniqueName: \"kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz\") pod \"737b7b5e-1188-4b76-9cea-fc60e57d4165\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") "
Nov 04 00:32:31.715799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.709067    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8\" (UniqueName: \"kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8\") pod \"295a683e-40ef-4e48-85b1-25f3d1603f52\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") "
Nov 04 00:32:31.723811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.716308    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume\") pod \"295a683e-40ef-4e48-85b1-25f3d1603f52\" (UID: \"295a683e-40ef-4e48-85b1-25f3d1603f52\") "
Nov 04 00:32:31.723811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.716486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume\") pod \"737b7b5e-1188-4b76-9cea-fc60e57d4165\" (UID: \"737b7b5e-1188-4b76-9cea-fc60e57d4165\") "
Nov 04 00:32:31.725514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.725109    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume" pod "737b7b5e-1188-4b76-9cea-fc60e57d4165" (UID: "737b7b5e-1188-4b76-9cea-fc60e57d4165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:31.771809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.732039    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume" pod "295a683e-40ef-4e48-85b1-25f3d1603f52" (UID: "295a683e-40ef-4e48-85b1-25f3d1603f52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:31.771809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.766457    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8" pod "295a683e-40ef-4e48-85b1-25f3d1603f52" (UID: "295a683e-40ef-4e48-85b1-25f3d1603f52"). InnerVolumeSpecName "kube-api-access-kq7d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:31.771809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.767941    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz" pod "737b7b5e-1188-4b76-9cea-fc60e57d4165" (UID: "737b7b5e-1188-4b76-9cea-fc60e57d4165"). InnerVolumeSpecName "kube-api-access-m7sbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:31.793398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.793361    5074 generic.go:356] "Generic (PLEG): container finished" podID="295a683e-40ef-4e48-85b1-25f3d1603f52" containerID="6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5" exitCode=0
Nov 04 00:32:31.793702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.793674    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb" event={"ID":"295a683e-40ef-4e48-85b1-25f3d1603f52","Type":"ContainerDied","Data":"6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5"}
Nov 04 00:32:31.793941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.793900    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb" event={"ID":"295a683e-40ef-4e48-85b1-25f3d1603f52","Type":"ContainerDied","Data":"bcfa13b8d8fd83a2ae9c39cce43aa4df05fb617f710c1de76a3370a663ce77cf"}
Nov 04 00:32:31.794105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.794088    5074 scope.go:122] "RemoveContainer" containerID="6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5"
Nov 04 00:32:31.794471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.794445    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"
Nov 04 00:32:31.820952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.818861    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7sbz\" (UniqueName: \"kubernetes.io/projected/737b7b5e-1188-4b76-9cea-fc60e57d4165-kube-api-access-m7sbz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:31.820952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.818893    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7d8\" (UniqueName: \"kubernetes.io/projected/295a683e-40ef-4e48-85b1-25f3d1603f52-kube-api-access-kq7d8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:31.820952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.818908    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295a683e-40ef-4e48-85b1-25f3d1603f52-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:31.820952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.818922    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737b7b5e-1188-4b76-9cea-fc60e57d4165-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:31.821717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.821680    5074 generic.go:356] "Generic (PLEG): container finished" podID="737b7b5e-1188-4b76-9cea-fc60e57d4165" containerID="2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f" exitCode=0
Nov 04 00:32:31.821902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.821880    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"
Nov 04 00:32:31.821988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.821969    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t" event={"ID":"737b7b5e-1188-4b76-9cea-fc60e57d4165","Type":"ContainerDied","Data":"2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f"}
Nov 04 00:32:31.822044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.822031    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t" event={"ID":"737b7b5e-1188-4b76-9cea-fc60e57d4165","Type":"ContainerDied","Data":"21da4f3cc10dbed2023c90aaa428b8eca94cf14bcb62954192ad0d9298c8d04e"}
Nov 04 00:32:31.828462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.828432    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p" event={"ID":"5385bcd5-a251-4445-9d38-b99d94c1b17e","Type":"ContainerStarted","Data":"b0f879181306758026bda3b860ffc1fa0e6e7d64321fdc0ff116b4fccaaa2bf5"}
Nov 04 00:32:31.828618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.828484    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p" event={"ID":"5385bcd5-a251-4445-9d38-b99d94c1b17e","Type":"ContainerStarted","Data":"34b4223b2ce236a26ce81e97ba8c94e62a59a5d2fd2dfa20ea00d24641586b77"}
Nov 04 00:32:31.834192 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.834171    5074 scope.go:122] "RemoveContainer" containerID="6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5"
Nov 04 00:32:31.834541 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:31.834516    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5\": container with ID starting with 6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5 not found: ID does not exist" containerID="6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5"
Nov 04 00:32:31.834643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.834555    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5"} err="failed to get container status \"6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5\": rpc error: code = NotFound desc = could not find container \"6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5\": container with ID starting with 6bf277555cf4ab8a58dcbb2eb67e5436a7ff5c23577903b5d12afe5afa8da3f5 not found: ID does not exist"
Nov 04 00:32:31.834643 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.834579    5074 scope.go:122] "RemoveContainer" containerID="2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f"
Nov 04 00:32:31.846426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.846403    5074 scope.go:122] "RemoveContainer" containerID="2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f"
Nov 04 00:32:31.846668 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:31.846653    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f\": container with ID starting with 2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f not found: ID does not exist" containerID="2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f"
Nov 04 00:32:31.846723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.846673    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f"} err="failed to get container status \"2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f\": rpc error: code = NotFound desc = could not find container \"2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f\": container with ID starting with 2bcbba8c701f29beb549d210e7a647a3b7a0a0d129c9c7e25b0c4c2b64c6fb5f not found: ID does not exist"
Nov 04 00:32:31.861459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.861406    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p" podStartSLOduration=1.861389785 podStartE2EDuration="1.861389785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:32:31.859208593 +0000 UTC m=+3816.958808737" watchObservedRunningTime="2026-11-04 00:32:31.861389785 +0000 UTC m=+3816.960989928"
Nov 04 00:32:31.877835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.877811    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"]
Nov 04 00:32:31.885338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.885303    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2341ca-57c6bddbc5-cf5zb"]
Nov 04 00:32:31.899231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.899202    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"]
Nov 04 00:32:31.904978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:31.904957    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2341ca-7744cb9d98-82t5t"]
Nov 04 00:32:32.843839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:32.843794    5074 generic.go:356] "Generic (PLEG): container finished" podID="5385bcd5-a251-4445-9d38-b99d94c1b17e" containerID="b0f879181306758026bda3b860ffc1fa0e6e7d64321fdc0ff116b4fccaaa2bf5" exitCode=0
Nov 04 00:32:32.844268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:32.843878    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p" event={"ID":"5385bcd5-a251-4445-9d38-b99d94c1b17e","Type":"ContainerDied","Data":"b0f879181306758026bda3b860ffc1fa0e6e7d64321fdc0ff116b4fccaaa2bf5"}
Nov 04 00:32:33.019940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:33.019899    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295a683e-40ef-4e48-85b1-25f3d1603f52" path="/var/lib/kubelet/pods/295a683e-40ef-4e48-85b1-25f3d1603f52/volumes"
Nov 04 00:32:33.020489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:33.020465    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737b7b5e-1188-4b76-9cea-fc60e57d4165" path="/var/lib/kubelet/pods/737b7b5e-1188-4b76-9cea-fc60e57d4165/volumes"
Nov 04 00:32:33.030165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:33.030139    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"]
Nov 04 00:32:33.033682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:33.033658    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffd8cfa-x6wsl"]
Nov 04 00:32:33.985303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:33.985237    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:32:34.047215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.047190    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:34.147298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.147220    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca\") pod \"5385bcd5-a251-4445-9d38-b99d94c1b17e\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") "
Nov 04 00:32:34.147298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.147277    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca\") pod \"5385bcd5-a251-4445-9d38-b99d94c1b17e\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") "
Nov 04 00:32:34.147484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.147365    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99\") pod \"5385bcd5-a251-4445-9d38-b99d94c1b17e\" (UID: \"5385bcd5-a251-4445-9d38-b99d94c1b17e\") "
Nov 04 00:32:34.147633 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.147608    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca" pod "5385bcd5-a251-4445-9d38-b99d94c1b17e" (UID: "5385bcd5-a251-4445-9d38-b99d94c1b17e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:34.147893 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.147876    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:34.149435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.149414    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca" pod "5385bcd5-a251-4445-9d38-b99d94c1b17e" (UID: "5385bcd5-a251-4445-9d38-b99d94c1b17e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:34.149532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.149509    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99" pod "5385bcd5-a251-4445-9d38-b99d94c1b17e" (UID: "5385bcd5-a251-4445-9d38-b99d94c1b17e"). InnerVolumeSpecName "kube-api-access-dct99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:34.248939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.248900    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5385bcd5-a251-4445-9d38-b99d94c1b17e-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:34.248939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.248928    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/5385bcd5-a251-4445-9d38-b99d94c1b17e-kube-api-access-dct99\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:34.866564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.866532    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"
Nov 04 00:32:34.866564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.866546    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p" event={"ID":"5385bcd5-a251-4445-9d38-b99d94c1b17e","Type":"ContainerDied","Data":"34b4223b2ce236a26ce81e97ba8c94e62a59a5d2fd2dfa20ea00d24641586b77"}
Nov 04 00:32:34.866852 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:34.866598    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b4223b2ce236a26ce81e97ba8c94e62a59a5d2fd2dfa20ea00d24641586b77"
Nov 04 00:32:35.020628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.020591    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc6fe8f-9a21-424a-95b0-04342455a5f8" path="/var/lib/kubelet/pods/fcc6fe8f-9a21-424a-95b0-04342455a5f8/volumes"
Nov 04 00:32:35.709454 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.709426    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:32:35.780203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.780098    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume\") pod \"85dbd136-817b-4a31-85c8-be66c017b81d\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") "
Nov 04 00:32:35.780402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.780254    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg\" (UniqueName: \"kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg\") pod \"85dbd136-817b-4a31-85c8-be66c017b81d\" (UID: \"85dbd136-817b-4a31-85c8-be66c017b81d\") "
Nov 04 00:32:35.780488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.780461    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume" pod "85dbd136-817b-4a31-85c8-be66c017b81d" (UID: "85dbd136-817b-4a31-85c8-be66c017b81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:35.780803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.780784    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dbd136-817b-4a31-85c8-be66c017b81d-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:35.782488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.782463    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg" pod "85dbd136-817b-4a31-85c8-be66c017b81d" (UID: "85dbd136-817b-4a31-85c8-be66c017b81d"). InnerVolumeSpecName "kube-api-access-229pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:35.876590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.876553    5074 generic.go:356] "Generic (PLEG): container finished" podID="85dbd136-817b-4a31-85c8-be66c017b81d" containerID="f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903" exitCode=0
Nov 04 00:32:35.876757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.876603    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"
Nov 04 00:32:35.876757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.876629    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" event={"ID":"85dbd136-817b-4a31-85c8-be66c017b81d","Type":"ContainerDied","Data":"f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903"}
Nov 04 00:32:35.876757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.876712    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd" event={"ID":"85dbd136-817b-4a31-85c8-be66c017b81d","Type":"ContainerDied","Data":"54c50e7ad7ecc36f010cf55ddb2d7be9d77e55ec0cc706973d36c69c2fc8ab30"}
Nov 04 00:32:35.876922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.876766    5074 scope.go:122] "RemoveContainer" containerID="f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903"
Nov 04 00:32:35.881731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.881707    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229pg\" (UniqueName: \"kubernetes.io/projected/85dbd136-817b-4a31-85c8-be66c017b81d-kube-api-access-229pg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:35.889331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.889312    5074 scope.go:122] "RemoveContainer" containerID="f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903"
Nov 04 00:32:35.889624 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:35.889598    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903\": container with ID starting with f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903 not found: ID does not exist" containerID="f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903"
Nov 04 00:32:35.889700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.889634    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903"} err="failed to get container status \"f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903\": rpc error: code = NotFound desc = could not find container \"f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903\": container with ID starting with f525c270ff80cce84957a1ce8c8e350abe05674645c8f3bc0344126e59027903 not found: ID does not exist"
Nov 04 00:32:35.905572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.905545    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"]
Nov 04 00:32:35.910648 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:35.910624    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7d218-df6b68579-9qftd"]
Nov 04 00:32:37.021373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:37.021333    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" path="/var/lib/kubelet/pods/85dbd136-817b-4a31-85c8-be66c017b81d/volumes"
Nov 04 00:32:39.103707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.103657    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main" containerID="cri-o://d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f" gracePeriod=45
Nov 04 00:32:39.414424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.414390    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 04 00:32:39.541163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541129    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir\") pod \"670271d0-1d7e-4041-97dd-82d07d99e5c0\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") "
Nov 04 00:32:39.541360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541194    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home\") pod \"670271d0-1d7e-4041-97dd-82d07d99e5c0\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") "
Nov 04 00:32:39.541360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541255    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs\") pod \"670271d0-1d7e-4041-97dd-82d07d99e5c0\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") "
Nov 04 00:32:39.541360 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541338    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq\" (UniqueName: \"kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq\") pod \"670271d0-1d7e-4041-97dd-82d07d99e5c0\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") "
Nov 04 00:32:39.541531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541380    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache\") pod \"670271d0-1d7e-4041-97dd-82d07d99e5c0\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") "
Nov 04 00:32:39.541531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541443    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm\") pod \"670271d0-1d7e-4041-97dd-82d07d99e5c0\" (UID: \"670271d0-1d7e-4041-97dd-82d07d99e5c0\") "
Nov 04 00:32:39.541531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541428    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir" pod "670271d0-1d7e-4041-97dd-82d07d99e5c0" (UID: "670271d0-1d7e-4041-97dd-82d07d99e5c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.541531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541499    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home" pod "670271d0-1d7e-4041-97dd-82d07d99e5c0" (UID: "670271d0-1d7e-4041-97dd-82d07d99e5c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.541832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.541685    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache" pod "670271d0-1d7e-4041-97dd-82d07d99e5c0" (UID: "670271d0-1d7e-4041-97dd-82d07d99e5c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.542248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.542224    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.542309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.542252    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.542309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.542266    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.547773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.544138    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm" pod "670271d0-1d7e-4041-97dd-82d07d99e5c0" (UID: "670271d0-1d7e-4041-97dd-82d07d99e5c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.547773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.544153    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs" pod "670271d0-1d7e-4041-97dd-82d07d99e5c0" (UID: "670271d0-1d7e-4041-97dd-82d07d99e5c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:39.547773 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.544264    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq" pod "670271d0-1d7e-4041-97dd-82d07d99e5c0" (UID: "670271d0-1d7e-4041-97dd-82d07d99e5c0"). InnerVolumeSpecName "kube-api-access-fz9fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:39.643434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.643340    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670271d0-1d7e-4041-97dd-82d07d99e5c0-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.643434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.643382    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9fq\" (UniqueName: \"kubernetes.io/projected/670271d0-1d7e-4041-97dd-82d07d99e5c0-kube-api-access-fz9fq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.643434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.643397    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670271d0-1d7e-4041-97dd-82d07d99e5c0-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.926891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.926777    5074 generic.go:356] "Generic (PLEG): container finished" podID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerID="d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f" exitCode=0
Nov 04 00:32:39.927082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.926922    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" event={"ID":"670271d0-1d7e-4041-97dd-82d07d99e5c0","Type":"ContainerDied","Data":"d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f"}
Nov 04 00:32:39.927082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.926961    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j" event={"ID":"670271d0-1d7e-4041-97dd-82d07d99e5c0","Type":"ContainerDied","Data":"bd04b4e759fb704bd32aeca268372634feb7d7c402349e6cbecf77fca69d6810"}
Nov 04 00:32:39.927082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.926983    5074 scope.go:122] "RemoveContainer" containerID="d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f"
Nov 04 00:32:39.927268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.927133    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"
Nov 04 00:32:39.942315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.942290    5074 scope.go:122] "RemoveContainer" containerID="d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f"
Nov 04 00:32:39.942626 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:39.942602    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f\": container with ID starting with d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f not found: ID does not exist" containerID="d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f"
Nov 04 00:32:39.942689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.942633    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f"} err="failed to get container status \"d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f\": rpc error: code = NotFound desc = could not find container \"d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f\": container with ID starting with d4e1d8ee86de4403269d1712a0c3902461a65402928daf48b34615617243193f not found: ID does not exist"
Nov 04 00:32:39.956005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.955980    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"]
Nov 04 00:32:39.959668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:39.959644    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/premium-w0-f6f292-kserve-9b758cb99-gml5j"]
Nov 04 00:32:41.022687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:41.022652    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" path="/var/lib/kubelet/pods/670271d0-1d7e-4041-97dd-82d07d99e5c0/volumes"
Nov 04 00:32:42.972144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:42.972108    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"]
Nov 04 00:32:42.972551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:42.972396    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" containerID="cri-o://ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5" gracePeriod=30
Nov 04 00:32:46.615921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:46.615885    5074 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:32:46.616403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:46.615940    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:32:48.268221 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.268197    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:32:48.373252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373221    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373314    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373340    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373377    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373416    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373486    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373530    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373553    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:48.373711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373578    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.373711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373685    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data\") pod \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\" (UID: \"fd2ed446-5d52-4b60-8125-8a28850bb5c5\") "
Nov 04 00:32:48.374132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.373806    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:48.374201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374157    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:48.374302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374275    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:48.374551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374527    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.374551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374553    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.374779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374567    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.374779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374579    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.375091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.374804    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:48.376356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.376325    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:32:48.376475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.376455    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:48.376602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.376578    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "kube-api-access-647hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:48.376989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.376967    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token" pod "fd2ed446-5d52-4b60-8125-8a28850bb5c5" (UID: "fd2ed446-5d52-4b60-8125-8a28850bb5c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:48.476278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.476202    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.476278 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.476274    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.476507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.476288    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647hf\" (UniqueName: \"kubernetes.io/projected/fd2ed446-5d52-4b60-8125-8a28850bb5c5-kube-api-access-647hf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.476507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.476304    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:48.476507 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:48.476316    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ed446-5d52-4b60-8125-8a28850bb5c5-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:49.043039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.042998    5074 generic.go:356] "Generic (PLEG): container finished" podID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerID="ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5" exitCode=0
Nov 04 00:32:49.043299 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.043284    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"
Nov 04 00:32:49.044264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.044236    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"]
Nov 04 00:32:49.044416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.044275    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" event={"ID":"fd2ed446-5d52-4b60-8125-8a28850bb5c5","Type":"ContainerDied","Data":"ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5"}
Nov 04 00:32:49.044416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.044318    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz" event={"ID":"fd2ed446-5d52-4b60-8125-8a28850bb5c5","Type":"ContainerDied","Data":"e2f99028d99e57b101ecfe9fef87f8e11df38f3446ec0c16fcb2b9b7e6e598b2"}
Nov 04 00:32:49.044416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.044347    5074 scope.go:122] "RemoveContainer" containerID="ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5"
Nov 04 00:32:49.047817 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.047793    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ad313b-l8bmw"]
Nov 04 00:32:49.063513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.063492    5074 scope.go:122] "RemoveContainer" containerID="ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5"
Nov 04 00:32:49.064711 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:49.064679    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5\": container with ID starting with ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5 not found: ID does not exist" containerID="ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5"
Nov 04 00:32:49.064847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.064706    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5"} err="failed to get container status \"ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5\": rpc error: code = NotFound desc = could not find container \"ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5\": container with ID starting with ab7293259c16d66f0c7d8255c96e7b381c9d62dcbb35ebdf30d8cedd6cc642d5 not found: ID does not exist"
Nov 04 00:32:49.078315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.078290    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"]
Nov 04 00:32:49.083012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:49.082991    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-xwvpz"]
Nov 04 00:32:51.021759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.021712    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fce481-441d-4b3f-93e3-67646d320b00" path="/var/lib/kubelet/pods/36fce481-441d-4b3f-93e3-67646d320b00/volumes"
Nov 04 00:32:51.022503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.022480    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" path="/var/lib/kubelet/pods/fd2ed446-5d52-4b60-8125-8a28850bb5c5/volumes"
Nov 04 00:32:51.417181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417099    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"]
Nov 04 00:32:51.417937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417916    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5385bcd5-a251-4445-9d38-b99d94c1b17e" containerName="revoke-keys"
Nov 04 00:32:51.417937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417937    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5385bcd5-a251-4445-9d38-b99d94c1b17e" containerName="revoke-keys"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417948    5074 cpu_manager.go:414] "containerMap: removing container" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" containerName="payload-pre-processing"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417953    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" containerName="payload-pre-processing"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417967    5074 cpu_manager.go:414] "containerMap: removing container" podUID="737b7b5e-1188-4b76-9cea-fc60e57d4165" containerName="payload-processing"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417973    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737b7b5e-1188-4b76-9cea-fc60e57d4165" containerName="payload-processing"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417985    5074 cpu_manager.go:414] "containerMap: removing container" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.417990    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418012    5074 cpu_manager.go:414] "containerMap: removing container" podUID="295a683e-40ef-4e48-85b1-25f3d1603f52" containerName="payload-pre-processing"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418017    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295a683e-40ef-4e48-85b1-25f3d1603f52" containerName="payload-pre-processing"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418029    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy"
Nov 04 00:32:51.418075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418033    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy"
Nov 04 00:32:51.418414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418140    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5385bcd5-a251-4445-9d38-b99d94c1b17e" containerName="revoke-keys"
Nov 04 00:32:51.418414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418150    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dbd136-817b-4a31-85c8-be66c017b81d" containerName="payload-pre-processing"
Nov 04 00:32:51.418414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418157    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="737b7b5e-1188-4b76-9cea-fc60e57d4165" containerName="payload-processing"
Nov 04 00:32:51.418414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418165    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="670271d0-1d7e-4041-97dd-82d07d99e5c0" containerName="main"
Nov 04 00:32:51.418414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418173    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="295a683e-40ef-4e48-85b1-25f3d1603f52" containerName="payload-pre-processing"
Nov 04 00:32:51.418414 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.418183    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2ed446-5d52-4b60-8125-8a28850bb5c5" containerName="istio-proxy"
Nov 04 00:32:51.423718 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.423693    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.429160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.429137    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"]
Nov 04 00:32:51.429915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.429886    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:32:51.430038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.429978    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:32:51.514061 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.514004    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwqx\" (UniqueName: \"kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.514274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.514209    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.514356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.514336    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.615860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.615809    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqx\" (UniqueName: \"kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.616074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.615986    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.616144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.616109    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.616942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.616913    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.619111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.619081    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.628422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.628390    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwqx\" (UniqueName: \"kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx\") pod \"maas-api-revoke-sub-08db5e8c-nvh2h\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:51.741310 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:51.741272    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:52.050549 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:52.050516    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"]
Nov 04 00:32:52.052189 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:32:52.052163    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df399a_0978_4e85_9de6_3e4cd6c21c9d.slice/crio-e4840a0abb01a6f383c87e863d3baaf95f1c11244fc39c70e781d30b9dec71d3 WatchSource:0}: Error finding container e4840a0abb01a6f383c87e863d3baaf95f1c11244fc39c70e781d30b9dec71d3: Status 404 returned error can't find the container with id e4840a0abb01a6f383c87e863d3baaf95f1c11244fc39c70e781d30b9dec71d3
Nov 04 00:32:52.071274 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:52.071084    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h" event={"ID":"e5df399a-0978-4e85-9de6-3e4cd6c21c9d","Type":"ContainerStarted","Data":"e4840a0abb01a6f383c87e863d3baaf95f1c11244fc39c70e781d30b9dec71d3"}
Nov 04 00:32:53.079796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:53.079759    5074 generic.go:356] "Generic (PLEG): container finished" podID="e5df399a-0978-4e85-9de6-3e4cd6c21c9d" containerID="2ba0056154771bfecf9d91f4c4db18af09b798bc0dabf25f372e1b1f5bb4825c" exitCode=0
Nov 04 00:32:53.080152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:53.079831    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h" event={"ID":"e5df399a-0978-4e85-9de6-3e4cd6c21c9d","Type":"ContainerDied","Data":"2ba0056154771bfecf9d91f4c4db18af09b798bc0dabf25f372e1b1f5bb4825c"}
Nov 04 00:32:54.271794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.271769    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:54.361519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.361425    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca\") pod \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") "
Nov 04 00:32:54.361699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.361518    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca\") pod \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") "
Nov 04 00:32:54.361699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.361587    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx\" (UniqueName: \"kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx\") pod \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\" (UID: \"e5df399a-0978-4e85-9de6-3e4cd6c21c9d\") "
Nov 04 00:32:54.361894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.361868    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca" pod "e5df399a-0978-4e85-9de6-3e4cd6c21c9d" (UID: "e5df399a-0978-4e85-9de6-3e4cd6c21c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:54.362134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.362116    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:54.363667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.363640    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca" pod "e5df399a-0978-4e85-9de6-3e4cd6c21c9d" (UID: "e5df399a-0978-4e85-9de6-3e4cd6c21c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:54.363667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.363650    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx" pod "e5df399a-0978-4e85-9de6-3e4cd6c21c9d" (UID: "e5df399a-0978-4e85-9de6-3e4cd6c21c9d"). InnerVolumeSpecName "kube-api-access-zvwqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:54.463077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.463044    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:54.463077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:54.463074    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwqx\" (UniqueName: \"kubernetes.io/projected/e5df399a-0978-4e85-9de6-3e4cd6c21c9d-kube-api-access-zvwqx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:55.096321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:55.096289    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"
Nov 04 00:32:55.096493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:55.096315    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h" event={"ID":"e5df399a-0978-4e85-9de6-3e4cd6c21c9d","Type":"ContainerDied","Data":"e4840a0abb01a6f383c87e863d3baaf95f1c11244fc39c70e781d30b9dec71d3"}
Nov 04 00:32:55.096493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:55.096362    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4840a0abb01a6f383c87e863d3baaf95f1c11244fc39c70e781d30b9dec71d3"
Nov 04 00:32:58.024765 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:32:58.024517    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:32:59.746425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:59.746377    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"]
Nov 04 00:32:59.753153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:32:59.753120    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"]
Nov 04 00:33:01.287514 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.287480    5074 scope.go:122] "RemoveContainer" containerID="93102fc8f1dcbe74380c6f6424d9bf5f947c5d619369a779d334c8b01bd33ad4"
Nov 04 00:33:01.298833 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.298801    5074 scope.go:122] "RemoveContainer" containerID="de46e6ae8347ebe05adab165e925c509391eb7bf6782d4ab85e9fefec82f6c06"
Nov 04 00:33:01.659692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.659603    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"]
Nov 04 00:33:01.660733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.660701    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e5df399a-0978-4e85-9de6-3e4cd6c21c9d" containerName="revoke-keys"
Nov 04 00:33:01.660733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.660732    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5df399a-0978-4e85-9de6-3e4cd6c21c9d" containerName="revoke-keys"
Nov 04 00:33:01.660969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.660954    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5df399a-0978-4e85-9de6-3e4cd6c21c9d" containerName="revoke-keys"
Nov 04 00:33:01.675285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.674684    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"]
Nov 04 00:33:01.675285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.674824    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.679837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.679811    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:33:01.683849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.681386    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:33:01.772891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.772857    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbqm\" (UniqueName: \"kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.773083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.772912    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.773211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.773179    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.874652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.874606    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbqm\" (UniqueName: \"kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.874843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.874697    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.874936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.874919    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.875776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.875731    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.877214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.877187    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:01.893638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:01.893458    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbqm\" (UniqueName: \"kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm\") pod \"maas-api-revoke-sub-5aae575d-gts9n\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:02.000566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:02.000526    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:02.321503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:02.321473    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"]
Nov 04 00:33:02.323331 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:33:02.323298    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1ae2d4_7398_4db8_bb7b_bfa8c36ebba4.slice/crio-e0c214f9bf3f434bf6d00c37a15d5416b773a875aece7a362716381b17c9c194 WatchSource:0}: Error finding container e0c214f9bf3f434bf6d00c37a15d5416b773a875aece7a362716381b17c9c194: Status 404 returned error can't find the container with id e0c214f9bf3f434bf6d00c37a15d5416b773a875aece7a362716381b17c9c194
Nov 04 00:33:03.176969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:03.176871    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n" event={"ID":"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4","Type":"ContainerStarted","Data":"0c020bfa185d346bfe41900845fd1e1d33b63c851afffdbfb0e0edb369f8cfe5"}
Nov 04 00:33:03.176969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:03.176952    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n" event={"ID":"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4","Type":"ContainerStarted","Data":"e0c214f9bf3f434bf6d00c37a15d5416b773a875aece7a362716381b17c9c194"}
Nov 04 00:33:03.199158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:03.199100    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n" podStartSLOduration=2.199083555 podStartE2EDuration="2.199083555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:33:03.198226129 +0000 UTC m=+3848.297826274" watchObservedRunningTime="2026-11-04 00:33:03.199083555 +0000 UTC m=+3848.298683700"
Nov 04 00:33:04.187586 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:04.187544    5074 generic.go:356] "Generic (PLEG): container finished" podID="9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" containerID="0c020bfa185d346bfe41900845fd1e1d33b63c851afffdbfb0e0edb369f8cfe5" exitCode=0
Nov 04 00:33:04.188135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:04.187584    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n" event={"ID":"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4","Type":"ContainerDied","Data":"0c020bfa185d346bfe41900845fd1e1d33b63c851afffdbfb0e0edb369f8cfe5"}
Nov 04 00:33:04.746755 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:04.746693    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z" podUID="11504999-0424-42d6-9585-03fb92e13ed2" containerName="payload-processing" containerID="cri-o://2cd21496c9939b4079f9acafb6538be0bf632fefabd2821fa9eb184b1dac9196" gracePeriod=25
Nov 04 00:33:04.754374 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:04.754302    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx" podUID="1e5a9dc6-78bc-4333-bf79-119ea7f8c331" containerName="payload-pre-processing" containerID="cri-o://b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab" gracePeriod=25
Nov 04 00:33:05.198211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.198178    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 04 00:33:05.247560 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.247524    5074 generic.go:356] "Generic (PLEG): container finished" podID="11504999-0424-42d6-9585-03fb92e13ed2" containerID="2cd21496c9939b4079f9acafb6538be0bf632fefabd2821fa9eb184b1dac9196" exitCode=0
Nov 04 00:33:05.247970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.247943    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z" event={"ID":"11504999-0424-42d6-9585-03fb92e13ed2","Type":"ContainerDied","Data":"2cd21496c9939b4079f9acafb6538be0bf632fefabd2821fa9eb184b1dac9196"}
Nov 04 00:33:05.248397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.248373    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6\" (UniqueName: \"kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6\") pod \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") "
Nov 04 00:33:05.248503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.248418    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume\") pod \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\" (UID: \"1e5a9dc6-78bc-4333-bf79-119ea7f8c331\") "
Nov 04 00:33:05.249243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.249205    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume" pod "1e5a9dc6-78bc-4333-bf79-119ea7f8c331" (UID: "1e5a9dc6-78bc-4333-bf79-119ea7f8c331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:05.250919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.250852    5074 generic.go:356] "Generic (PLEG): container finished" podID="1e5a9dc6-78bc-4333-bf79-119ea7f8c331" containerID="b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab" exitCode=0
Nov 04 00:33:05.251230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.251212    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"
Nov 04 00:33:05.251573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.251554    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx" event={"ID":"1e5a9dc6-78bc-4333-bf79-119ea7f8c331","Type":"ContainerDied","Data":"b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab"}
Nov 04 00:33:05.251705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.251598    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx" event={"ID":"1e5a9dc6-78bc-4333-bf79-119ea7f8c331","Type":"ContainerDied","Data":"7973230355f8a60017aecc768b6511265523d9a682592fbdfe2c16e8c3a84ccd"}
Nov 04 00:33:05.251705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.251640    5074 scope.go:122] "RemoveContainer" containerID="b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab"
Nov 04 00:33:05.263477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.263366    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 04 00:33:05.283613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.283513    5074 scope.go:122] "RemoveContainer" containerID="b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab"
Nov 04 00:33:05.288043 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:05.286150    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab\": container with ID starting with b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab not found: ID does not exist" containerID="b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab"
Nov 04 00:33:05.288043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.286191    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab"} err="failed to get container status \"b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab\": rpc error: code = NotFound desc = could not find container \"b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab\": container with ID starting with b869dce63acb38803b72ddac8479261240010fcb2e0e400c3ee381274675c2ab not found: ID does not exist"
Nov 04 00:33:05.289086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.288227    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6" pod "1e5a9dc6-78bc-4333-bf79-119ea7f8c331" (UID: "1e5a9dc6-78bc-4333-bf79-119ea7f8c331"). InnerVolumeSpecName "kube-api-access-xgmk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:05.350349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.350310    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume\") pod \"11504999-0424-42d6-9585-03fb92e13ed2\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") "
Nov 04 00:33:05.350626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.350610    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9\" (UniqueName: \"kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9\") pod \"11504999-0424-42d6-9585-03fb92e13ed2\" (UID: \"11504999-0424-42d6-9585-03fb92e13ed2\") "
Nov 04 00:33:05.351218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.351197    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmk6\" (UniqueName: \"kubernetes.io/projected/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-kube-api-access-xgmk6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:05.351263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.351224    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5a9dc6-78bc-4333-bf79-119ea7f8c331-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:05.352007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.351980    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume" pod "11504999-0424-42d6-9585-03fb92e13ed2" (UID: "11504999-0424-42d6-9585-03fb92e13ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:05.353566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.353535    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9" pod "11504999-0424-42d6-9585-03fb92e13ed2" (UID: "11504999-0424-42d6-9585-03fb92e13ed2"). InnerVolumeSpecName "kube-api-access-m8rn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:05.452969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.452929    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rn9\" (UniqueName: \"kubernetes.io/projected/11504999-0424-42d6-9585-03fb92e13ed2-kube-api-access-m8rn9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:05.452969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.452966    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11504999-0424-42d6-9585-03fb92e13ed2-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:05.472530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.472500    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:05.553933 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.553845    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca\") pod \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") "
Nov 04 00:33:05.554296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.554265    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca" pod "9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" (UID: "9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:05.554418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.554282    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm\" (UniqueName: \"kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm\") pod \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") "
Nov 04 00:33:05.554418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.554343    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca\") pod \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\" (UID: \"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4\") "
Nov 04 00:33:05.554910 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.554882    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:05.557216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.557192    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm" pod "9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" (UID: "9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4"). InnerVolumeSpecName "kube-api-access-7gbqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:05.558889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.558867    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca" pod "9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" (UID: "9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:05.579860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.579836    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"]
Nov 04 00:33:05.585100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.585039    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2b1f66-7c88bd949f-dbfnx"]
Nov 04 00:33:05.656287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.656209    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gbqm\" (UniqueName: \"kubernetes.io/projected/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-kube-api-access-7gbqm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:05.656287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:05.656245    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:06.259381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.259346    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"
Nov 04 00:33:06.259821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.259382    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n" event={"ID":"9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4","Type":"ContainerDied","Data":"e0c214f9bf3f434bf6d00c37a15d5416b773a875aece7a362716381b17c9c194"}
Nov 04 00:33:06.259821 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.259430    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c214f9bf3f434bf6d00c37a15d5416b773a875aece7a362716381b17c9c194"
Nov 04 00:33:06.261039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.261017    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"
Nov 04 00:33:06.261165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.261045    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z" event={"ID":"11504999-0424-42d6-9585-03fb92e13ed2","Type":"ContainerDied","Data":"fb492a527d9de593a953004c5dc46462aa4ede034194597b6b0351a187bc4cbc"}
Nov 04 00:33:06.261165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.261102    5074 scope.go:122] "RemoveContainer" containerID="2cd21496c9939b4079f9acafb6538be0bf632fefabd2821fa9eb184b1dac9196"
Nov 04 00:33:06.291458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.291425    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"]
Nov 04 00:33:06.295922 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:06.295896    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2b1f66-8645d5786b-8bq6z"]
Nov 04 00:33:07.020363 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:07.020320    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11504999-0424-42d6-9585-03fb92e13ed2" path="/var/lib/kubelet/pods/11504999-0424-42d6-9585-03fb92e13ed2/volumes"
Nov 04 00:33:07.020832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:07.020810    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5a9dc6-78bc-4333-bf79-119ea7f8c331" path="/var/lib/kubelet/pods/1e5a9dc6-78bc-4333-bf79-119ea7f8c331/volumes"
Nov 04 00:33:12.376461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.376421    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"]
Nov 04 00:33:12.377111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377092    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1e5a9dc6-78bc-4333-bf79-119ea7f8c331" containerName="payload-pre-processing"
Nov 04 00:33:12.377178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377111    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5a9dc6-78bc-4333-bf79-119ea7f8c331" containerName="payload-pre-processing"
Nov 04 00:33:12.377178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377152    5074 cpu_manager.go:414] "containerMap: removing container" podUID="11504999-0424-42d6-9585-03fb92e13ed2" containerName="payload-processing"
Nov 04 00:33:12.377178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377158    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11504999-0424-42d6-9585-03fb92e13ed2" containerName="payload-processing"
Nov 04 00:33:12.377178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377174    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" containerName="revoke-keys"
Nov 04 00:33:12.377178 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377178    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" containerName="revoke-keys"
Nov 04 00:33:12.377440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377286    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11504999-0424-42d6-9585-03fb92e13ed2" containerName="payload-processing"
Nov 04 00:33:12.377440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377297    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" containerName="revoke-keys"
Nov 04 00:33:12.377440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.377306    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5a9dc6-78bc-4333-bf79-119ea7f8c331" containerName="payload-pre-processing"
Nov 04 00:33:12.381168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.381148    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.391720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.391692    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"]
Nov 04 00:33:12.391934 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.391911    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:33:12.392132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.392113    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:33:12.564994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.564952    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.565154 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.565033    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8sn\" (UniqueName: \"kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.565154 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.565137    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.600735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.600694    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"]
Nov 04 00:33:12.601018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.600972    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" containerID="cri-o://18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d" gracePeriod=30
Nov 04 00:33:12.667049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.666965    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8sn\" (UniqueName: \"kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.667049 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.667032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.667272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.667188    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.667943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.667911    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.669520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.669489    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.680008 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.679978    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8sn\" (UniqueName: \"kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn\") pod \"maas-api-revoke-sub-24f46a44-dq2pt\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:12.698587 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:12.698556    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:13.056928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:13.056898    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"]
Nov 04 00:33:13.057440 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:33:13.057413    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be6b639_36f2_4530_888a_06cc6e0a2291.slice/crio-da62ab259098b79debac2e64946cf422a544aedd253f63045ca89286bc3845bc WatchSource:0}: Error finding container da62ab259098b79debac2e64946cf422a544aedd253f63045ca89286bc3845bc: Status 404 returned error can't find the container with id da62ab259098b79debac2e64946cf422a544aedd253f63045ca89286bc3845bc
Nov 04 00:33:13.323555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:13.323513    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt" event={"ID":"5be6b639-36f2-4530-888a-06cc6e0a2291","Type":"ContainerStarted","Data":"da62ab259098b79debac2e64946cf422a544aedd253f63045ca89286bc3845bc"}
Nov 04 00:33:13.387486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:13.387455    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"]
Nov 04 00:33:13.387964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:13.387635    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy" containerID="cri-o://54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78" gracePeriod=30
Nov 04 00:33:14.335420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:14.335386    5074 generic.go:356] "Generic (PLEG): container finished" podID="5be6b639-36f2-4530-888a-06cc6e0a2291" containerID="a0751fa379b62fb831b6b86692bc7c7eb2d37f76d48642f5ab47331d2a3b477e" exitCode=0
Nov 04 00:33:14.335628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:14.335472    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt" event={"ID":"5be6b639-36f2-4530-888a-06cc6e0a2291","Type":"ContainerDied","Data":"a0751fa379b62fb831b6b86692bc7c7eb2d37f76d48642f5ab47331d2a3b477e"}
Nov 04 00:33:15.539872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.539721    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:15.623319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.623224    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn\" (UniqueName: \"kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn\") pod \"5be6b639-36f2-4530-888a-06cc6e0a2291\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") "
Nov 04 00:33:15.623645 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.623616    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca\") pod \"5be6b639-36f2-4530-888a-06cc6e0a2291\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") "
Nov 04 00:33:15.623884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.623862    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca\") pod \"5be6b639-36f2-4530-888a-06cc6e0a2291\" (UID: \"5be6b639-36f2-4530-888a-06cc6e0a2291\") "
Nov 04 00:33:15.625258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.625232    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca" pod "5be6b639-36f2-4530-888a-06cc6e0a2291" (UID: "5be6b639-36f2-4530-888a-06cc6e0a2291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:15.641767 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.640930    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca" pod "5be6b639-36f2-4530-888a-06cc6e0a2291" (UID: "5be6b639-36f2-4530-888a-06cc6e0a2291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:15.646436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.646405    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn" pod "5be6b639-36f2-4530-888a-06cc6e0a2291" (UID: "5be6b639-36f2-4530-888a-06cc6e0a2291"). InnerVolumeSpecName "kube-api-access-pn8sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:15.729759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.726717    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:15.729759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.726776    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8sn\" (UniqueName: \"kubernetes.io/projected/5be6b639-36f2-4530-888a-06cc6e0a2291-kube-api-access-pn8sn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:15.729759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:15.726793    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be6b639-36f2-4530-888a-06cc6e0a2291-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:16.359484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.359447    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"
Nov 04 00:33:16.359484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.359467    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt" event={"ID":"5be6b639-36f2-4530-888a-06cc6e0a2291","Type":"ContainerDied","Data":"da62ab259098b79debac2e64946cf422a544aedd253f63045ca89286bc3845bc"}
Nov 04 00:33:16.359721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.359518    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da62ab259098b79debac2e64946cf422a544aedd253f63045ca89286bc3845bc"
Nov 04 00:33:16.511618 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.511581    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"]
Nov 04 00:33:16.512669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.512648    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5be6b639-36f2-4530-888a-06cc6e0a2291" containerName="revoke-keys"
Nov 04 00:33:16.512734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.512672    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be6b639-36f2-4530-888a-06cc6e0a2291" containerName="revoke-keys"
Nov 04 00:33:16.512877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.512864    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be6b639-36f2-4530-888a-06cc6e0a2291" containerName="revoke-keys"
Nov 04 00:33:16.517777 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.517228    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.522758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.522435    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:33:16.522758 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.522519    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:33:16.523109 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.523083    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"]
Nov 04 00:33:16.540795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.540724    5074 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:33:16.541283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.541253    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:33:16.641074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.640980    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.641074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.641047    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.641332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.641305    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4s9\" (UniqueName: \"kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.743135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.743097    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4s9\" (UniqueName: \"kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.743348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.743252    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.743348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.743298    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.743998 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.743974    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.745890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.745867    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.756105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.756085    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4s9\" (UniqueName: \"kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9\") pod \"maas-api-revoke-sub-1ce365dd-z8nzn\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:16.844584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:16.844542    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:17.142701 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:33:17.142664    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058ec553_4725_43de_aeb2_08c4df5c4f56.slice/crio-38284045e2bdac25a1a66b049bc356c8dfd55468f58bad1bfc8788e845f69a15 WatchSource:0}: Error finding container 38284045e2bdac25a1a66b049bc356c8dfd55468f58bad1bfc8788e845f69a15: Status 404 returned error can't find the container with id 38284045e2bdac25a1a66b049bc356c8dfd55468f58bad1bfc8788e845f69a15
Nov 04 00:33:17.143692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.143661    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"]
Nov 04 00:33:17.368352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.368316    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn" event={"ID":"058ec553-4725-43de-aeb2-08c4df5c4f56","Type":"ContainerStarted","Data":"38284045e2bdac25a1a66b049bc356c8dfd55468f58bad1bfc8788e845f69a15"}
Nov 04 00:33:17.913172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.913138    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 04 00:33:17.960077 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960043    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960178    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960216    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960245    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960325    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960437    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960491    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960568    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960610    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs\") pod \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\" (UID: \"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc\") "
Nov 04 00:33:17.960695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960658    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:17.960876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960822    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:17.961305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.960980    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:17.961305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961109    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:17.961305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961210    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:17.961552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961315    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:17.961552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961338    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:17.961552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961351    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:17.961552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961365    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:17.961552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.961378    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:17.962881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.962852    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:33:17.963375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.963348    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:17.964341 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.964283    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "kube-api-access-tjhdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:17.965168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:17.965138    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token" pod "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" (UID: "7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:18.062599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.062568    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.062599 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.062597    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.062823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.062607    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhdp\" (UniqueName: \"kubernetes.io/projected/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-kube-api-access-tjhdp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.062823 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.062616    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.379074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.378976    5074 generic.go:356] "Generic (PLEG): container finished" podID="058ec553-4725-43de-aeb2-08c4df5c4f56" containerID="5ff4ae306a53cd668f428eb10525ccebbf39b9a1a14d0c4c498da4dd91626ba0" exitCode=0
Nov 04 00:33:18.379236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.379068    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn" event={"ID":"058ec553-4725-43de-aeb2-08c4df5c4f56","Type":"ContainerDied","Data":"5ff4ae306a53cd668f428eb10525ccebbf39b9a1a14d0c4c498da4dd91626ba0"}
Nov 04 00:33:18.380717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.380693    5074 generic.go:356] "Generic (PLEG): container finished" podID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerID="18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d" exitCode=0
Nov 04 00:33:18.380858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.380772    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"
Nov 04 00:33:18.380858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.380784    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" event={"ID":"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc","Type":"ContainerDied","Data":"18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d"}
Nov 04 00:33:18.380858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.380831    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw" event={"ID":"7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc","Type":"ContainerDied","Data":"b3cdc55ae5a6797bd2a9fa0324ac7f97c9859d79b7d1c85ebe12a314022f635a"}
Nov 04 00:33:18.380996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.380852    5074 scope.go:122] "RemoveContainer" containerID="18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d"
Nov 04 00:33:18.396721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.396696    5074 scope.go:122] "RemoveContainer" containerID="18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d"
Nov 04 00:33:18.397086 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:18.397010    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d\": container with ID starting with 18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d not found: ID does not exist" containerID="18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d"
Nov 04 00:33:18.397086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.397049    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d"} err="failed to get container status \"18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d\": rpc error: code = NotFound desc = could not find container \"18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d\": container with ID starting with 18c79d20d13bb373af4eea8a2c325475e688d58822be7ccf4e311c94061c975d not found: ID does not exist"
Nov 04 00:33:18.422478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.422448    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"]
Nov 04 00:33:18.427881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.427858    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2skbw"]
Nov 04 00:33:18.657921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.657895    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 04 00:33:18.771306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771277    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771342    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771382    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771483    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771576 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771529    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771697    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771775    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771705    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:18.771839 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771791    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:18.771938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771850    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771894    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket\") pod \"8486895c-4686-4f0e-9286-f74f6303ab83\" (UID: \"8486895c-4686-4f0e-9286-f74f6303ab83\") "
Nov 04 00:33:18.771938 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.771899    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:18.772215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772184    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:18.772288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772190    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:18.772533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772504    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.772650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772538    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.772650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772555    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8486895c-4686-4f0e-9286-f74f6303ab83-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.772650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772570    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.772650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.772585    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.773722 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.773696    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "kube-api-access-6ncrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:18.773876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.773851    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:33:18.774102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.774078    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:18.774194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.774103    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy" pod "8486895c-4686-4f0e-9286-f74f6303ab83" (UID: "8486895c-4686-4f0e-9286-f74f6303ab83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:18.874018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.873974    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.874018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.874010    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8486895c-4686-4f0e-9286-f74f6303ab83-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.874018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.874022    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/8486895c-4686-4f0e-9286-f74f6303ab83-kube-api-access-6ncrg\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:18.874265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:18.874032    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8486895c-4686-4f0e-9286-f74f6303ab83-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:19.020259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.020220    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" path="/var/lib/kubelet/pods/7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc/volumes"
Nov 04 00:33:19.390203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.390117    5074 generic.go:356] "Generic (PLEG): container finished" podID="8486895c-4686-4f0e-9286-f74f6303ab83" containerID="54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78" exitCode=0
Nov 04 00:33:19.390203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.390176    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"
Nov 04 00:33:19.390422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.390201    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" event={"ID":"8486895c-4686-4f0e-9286-f74f6303ab83","Type":"ContainerDied","Data":"54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78"}
Nov 04 00:33:19.390422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.390256    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv" event={"ID":"8486895c-4686-4f0e-9286-f74f6303ab83","Type":"ContainerDied","Data":"a10c294292e86d2caf15e2f5637d915082872ede5333491383c0c43264c3f788"}
Nov 04 00:33:19.390422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.390285    5074 scope.go:122] "RemoveContainer" containerID="54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78"
Nov 04 00:33:19.419552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.419512    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"]
Nov 04 00:33:19.426033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.425578    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-kltjv"]
Nov 04 00:33:19.430058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.430034    5074 scope.go:122] "RemoveContainer" containerID="54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78"
Nov 04 00:33:19.430439 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:19.430412    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78\": container with ID starting with 54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78 not found: ID does not exist" containerID="54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78"
Nov 04 00:33:19.430530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.430451    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78"} err="failed to get container status \"54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78\": rpc error: code = NotFound desc = could not find container \"54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78\": container with ID starting with 54c2a0aa0f7059437ce658c71e69c8f5714c1434b430663b5d399c342e206a78 not found: ID does not exist"
Nov 04 00:33:19.564604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.564579    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:19.682538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.682458    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9\" (UniqueName: \"kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9\") pod \"058ec553-4725-43de-aeb2-08c4df5c4f56\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") "
Nov 04 00:33:19.682681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.682626    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca\") pod \"058ec553-4725-43de-aeb2-08c4df5c4f56\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") "
Nov 04 00:33:19.682797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.682778    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca\") pod \"058ec553-4725-43de-aeb2-08c4df5c4f56\" (UID: \"058ec553-4725-43de-aeb2-08c4df5c4f56\") "
Nov 04 00:33:19.683131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.683101    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca" pod "058ec553-4725-43de-aeb2-08c4df5c4f56" (UID: "058ec553-4725-43de-aeb2-08c4df5c4f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:19.683294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.683282    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:19.684622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.684599    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9" pod "058ec553-4725-43de-aeb2-08c4df5c4f56" (UID: "058ec553-4725-43de-aeb2-08c4df5c4f56"). InnerVolumeSpecName "kube-api-access-dz4s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:19.684730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.684654    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca" pod "058ec553-4725-43de-aeb2-08c4df5c4f56" (UID: "058ec553-4725-43de-aeb2-08c4df5c4f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:19.784230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.784197    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4s9\" (UniqueName: \"kubernetes.io/projected/058ec553-4725-43de-aeb2-08c4df5c4f56-kube-api-access-dz4s9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:19.784230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:19.784228    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058ec553-4725-43de-aeb2-08c4df5c4f56-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:20.400870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:20.400830    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn" event={"ID":"058ec553-4725-43de-aeb2-08c4df5c4f56","Type":"ContainerDied","Data":"38284045e2bdac25a1a66b049bc356c8dfd55468f58bad1bfc8788e845f69a15"}
Nov 04 00:33:20.400870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:20.400865    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"
Nov 04 00:33:20.401301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:20.400885    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38284045e2bdac25a1a66b049bc356c8dfd55468f58bad1bfc8788e845f69a15"
Nov 04 00:33:21.021308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:21.021266    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" path="/var/lib/kubelet/pods/8486895c-4686-4f0e-9286-f74f6303ab83/volumes"
Nov 04 00:33:21.083351 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:21.083310    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"]
Nov 04 00:33:21.091687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:21.091655    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"]
Nov 04 00:33:22.785399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.785357    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"]
Nov 04 00:33:22.786246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786220    5074 cpu_manager.go:414] "containerMap: removing container" podUID="058ec553-4725-43de-aeb2-08c4df5c4f56" containerName="revoke-keys"
Nov 04 00:33:22.786246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786246    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058ec553-4725-43de-aeb2-08c4df5c4f56" containerName="revoke-keys"
Nov 04 00:33:22.786423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786269    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy"
Nov 04 00:33:22.786423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786277    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy"
Nov 04 00:33:22.786423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786308    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy"
Nov 04 00:33:22.786423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786316    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy"
Nov 04 00:33:22.786635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786512    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8486895c-4686-4f0e-9286-f74f6303ab83" containerName="istio-proxy"
Nov 04 00:33:22.786635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786536    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="058ec553-4725-43de-aeb2-08c4df5c4f56" containerName="revoke-keys"
Nov 04 00:33:22.786635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.786546    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa649ec-dbbe-4b98-8287-da4b1f7e7fdc" containerName="istio-proxy"
Nov 04 00:33:22.791595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.791571    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:22.797203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.797176    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:33:22.797317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.797244    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:33:22.805964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.805938    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"]
Nov 04 00:33:22.932530 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.932488    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:22.932687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.932653    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkj5m\" (UniqueName: \"kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:22.932786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:22.932765    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.034553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.034509    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkj5m\" (UniqueName: \"kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.034731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.034592    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.034731 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.034691    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.035389 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.035366    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.037227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.037179    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.047298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.047277    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkj5m\" (UniqueName: \"kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m\") pod \"maas-api-revoke-sub-ae99f35d-q9hhm\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.117313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.117285    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:23.409317 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.409286    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"]
Nov 04 00:33:23.410538 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:33:23.410504    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc998e42_fbff_484a_9736_0ced8d7afec3.slice/crio-cf57a3a7404a3e07f600c3bb63f2fda3789133915f5f9d276a143facb73a1118 WatchSource:0}: Error finding container cf57a3a7404a3e07f600c3bb63f2fda3789133915f5f9d276a143facb73a1118: Status 404 returned error can't find the container with id cf57a3a7404a3e07f600c3bb63f2fda3789133915f5f9d276a143facb73a1118
Nov 04 00:33:23.428808 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:23.428781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm" event={"ID":"dc998e42-fbff-484a-9736-0ced8d7afec3","Type":"ContainerStarted","Data":"cf57a3a7404a3e07f600c3bb63f2fda3789133915f5f9d276a143facb73a1118"}
Nov 04 00:33:24.440843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:24.440799    5074 generic.go:356] "Generic (PLEG): container finished" podID="dc998e42-fbff-484a-9736-0ced8d7afec3" containerID="a1831c64dc092e1cfcb1d69d60e50f2042117469090182994c06878ab08e87c5" exitCode=0
Nov 04 00:33:24.441319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:24.440896    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm" event={"ID":"dc998e42-fbff-484a-9736-0ced8d7afec3","Type":"ContainerDied","Data":"a1831c64dc092e1cfcb1d69d60e50f2042117469090182994c06878ab08e87c5"}
Nov 04 00:33:25.621397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.621369    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:25.769932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.769833    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca\") pod \"dc998e42-fbff-484a-9736-0ced8d7afec3\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") "
Nov 04 00:33:25.770108 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.770082    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m\" (UniqueName: \"kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m\") pod \"dc998e42-fbff-484a-9736-0ced8d7afec3\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") "
Nov 04 00:33:25.770165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.770115    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca\") pod \"dc998e42-fbff-484a-9736-0ced8d7afec3\" (UID: \"dc998e42-fbff-484a-9736-0ced8d7afec3\") "
Nov 04 00:33:25.770547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.770514    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca" pod "dc998e42-fbff-484a-9736-0ced8d7afec3" (UID: "dc998e42-fbff-484a-9736-0ced8d7afec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:25.770719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.770659    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:25.772042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.772017    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca" pod "dc998e42-fbff-484a-9736-0ced8d7afec3" (UID: "dc998e42-fbff-484a-9736-0ced8d7afec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:25.772227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.772207    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m" pod "dc998e42-fbff-484a-9736-0ced8d7afec3" (UID: "dc998e42-fbff-484a-9736-0ced8d7afec3"). InnerVolumeSpecName "kube-api-access-rkj5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:25.872042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.872004    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkj5m\" (UniqueName: \"kubernetes.io/projected/dc998e42-fbff-484a-9736-0ced8d7afec3-kube-api-access-rkj5m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:25.872042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:25.872036    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc998e42-fbff-484a-9736-0ced8d7afec3-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:26.459558 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:26.459528    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"
Nov 04 00:33:26.459766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:26.459569    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm" event={"ID":"dc998e42-fbff-484a-9736-0ced8d7afec3","Type":"ContainerDied","Data":"cf57a3a7404a3e07f600c3bb63f2fda3789133915f5f9d276a143facb73a1118"}
Nov 04 00:33:26.459766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:26.459615    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf57a3a7404a3e07f600c3bb63f2fda3789133915f5f9d276a143facb73a1118"
Nov 04 00:33:27.296610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:27.296578    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"]
Nov 04 00:33:27.313131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:27.313098    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"]
Nov 04 00:33:28.032203 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:28.032158    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:33:30.252348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.252309    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"]
Nov 04 00:33:30.253165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.253144    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dc998e42-fbff-484a-9736-0ced8d7afec3" containerName="revoke-keys"
Nov 04 00:33:30.253229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.253167    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc998e42-fbff-484a-9736-0ced8d7afec3" containerName="revoke-keys"
Nov 04 00:33:30.253287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.253277    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc998e42-fbff-484a-9736-0ced8d7afec3" containerName="revoke-keys"
Nov 04 00:33:30.256850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.256822    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.265419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.265394    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"]
Nov 04 00:33:30.323672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.323637    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:33:30.323865 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.323711    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:33:30.329689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.329649    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.329861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.329697    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.329966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.329936    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcs89\" (UniqueName: \"kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.431290 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.431254    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcs89\" (UniqueName: \"kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.431504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.431432    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.431504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.431473    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.432384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.432349    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.434088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.434058    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.444330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.444291    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcs89\" (UniqueName: \"kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89\") pod \"maas-api-revoke-sub-2f43b351-lth8f\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.575943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.575823    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:30.872270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:30.872240    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"]
Nov 04 00:33:30.874021 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:33:30.873985    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1c2495_9587_48c0_ba35_2be588219891.slice/crio-c8ff0c322214717e86c6a8ea4bf659f8bd698629ff0ebf46099797790fb88dad WatchSource:0}: Error finding container c8ff0c322214717e86c6a8ea4bf659f8bd698629ff0ebf46099797790fb88dad: Status 404 returned error can't find the container with id c8ff0c322214717e86c6a8ea4bf659f8bd698629ff0ebf46099797790fb88dad
Nov 04 00:33:31.498333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:31.498233    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f" event={"ID":"1e1c2495-9587-48c0-ba35-2be588219891","Type":"ContainerStarted","Data":"6e26b63cf155b46a1dc4272ddc18260ca9b1fe157cf7e621811a4aa15259856c"}
Nov 04 00:33:31.498843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:31.498337    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f" event={"ID":"1e1c2495-9587-48c0-ba35-2be588219891","Type":"ContainerStarted","Data":"c8ff0c322214717e86c6a8ea4bf659f8bd698629ff0ebf46099797790fb88dad"}
Nov 04 00:33:31.517658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:31.517599    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f" podStartSLOduration=1.517583573 podStartE2EDuration="1.517583573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:33:31.517528529 +0000 UTC m=+3876.617128702" watchObservedRunningTime="2026-11-04 00:33:31.517583573 +0000 UTC m=+3876.617183717"
Nov 04 00:33:32.297074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.297012    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l" podUID="89909b81-d1a4-4134-9692-9544cba7903b" containerName="payload-processing" containerID="cri-o://de875d08255e379f7e1bd5214cce074e104d7e074563c1c0a6997a7c51a4d96c" gracePeriod=25
Nov 04 00:33:32.314590 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.314523    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7" podUID="a27b72ea-653a-47e3-9330-312df17c16cb" containerName="payload-pre-processing" containerID="cri-o://8e0efaf924feaf67f7a049b4faf8105462bd94d96bfc62e2d7549f1063671a3f" gracePeriod=25
Nov 04 00:33:32.532312 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.532271    5074 generic.go:356] "Generic (PLEG): container finished" podID="89909b81-d1a4-4134-9692-9544cba7903b" containerID="de875d08255e379f7e1bd5214cce074e104d7e074563c1c0a6997a7c51a4d96c" exitCode=0
Nov 04 00:33:32.532826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.532346    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l" event={"ID":"89909b81-d1a4-4134-9692-9544cba7903b","Type":"ContainerDied","Data":"de875d08255e379f7e1bd5214cce074e104d7e074563c1c0a6997a7c51a4d96c"}
Nov 04 00:33:32.534379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.534354    5074 generic.go:356] "Generic (PLEG): container finished" podID="1e1c2495-9587-48c0-ba35-2be588219891" containerID="6e26b63cf155b46a1dc4272ddc18260ca9b1fe157cf7e621811a4aa15259856c" exitCode=0
Nov 04 00:33:32.534789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.534435    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f" event={"ID":"1e1c2495-9587-48c0-ba35-2be588219891","Type":"ContainerDied","Data":"6e26b63cf155b46a1dc4272ddc18260ca9b1fe157cf7e621811a4aa15259856c"}
Nov 04 00:33:32.536461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.536440    5074 generic.go:356] "Generic (PLEG): container finished" podID="a27b72ea-653a-47e3-9330-312df17c16cb" containerID="8e0efaf924feaf67f7a049b4faf8105462bd94d96bfc62e2d7549f1063671a3f" exitCode=0
Nov 04 00:33:32.536573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.536510    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7" event={"ID":"a27b72ea-653a-47e3-9330-312df17c16cb","Type":"ContainerDied","Data":"8e0efaf924feaf67f7a049b4faf8105462bd94d96bfc62e2d7549f1063671a3f"}
Nov 04 00:33:32.618447 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.618415    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 04 00:33:32.665776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.665346    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume" pod "89909b81-d1a4-4134-9692-9544cba7903b" (UID: "89909b81-d1a4-4134-9692-9544cba7903b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:32.665776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.664839    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume\") pod \"89909b81-d1a4-4134-9692-9544cba7903b\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") "
Nov 04 00:33:32.666031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.665941    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6\" (UniqueName: \"kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6\") pod \"89909b81-d1a4-4134-9692-9544cba7903b\" (UID: \"89909b81-d1a4-4134-9692-9544cba7903b\") "
Nov 04 00:33:32.669034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.667377    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89909b81-d1a4-4134-9692-9544cba7903b-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:32.678764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.677884    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6" pod "89909b81-d1a4-4134-9692-9544cba7903b" (UID: "89909b81-d1a4-4134-9692-9544cba7903b"). InnerVolumeSpecName "kube-api-access-krfs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:32.698788 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.698761    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 04 00:33:32.768840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.768800    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp\" (UniqueName: \"kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp\") pod \"a27b72ea-653a-47e3-9330-312df17c16cb\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") "
Nov 04 00:33:32.769047 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.768988    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume\") pod \"a27b72ea-653a-47e3-9330-312df17c16cb\" (UID: \"a27b72ea-653a-47e3-9330-312df17c16cb\") "
Nov 04 00:33:32.769279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.769263    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krfs6\" (UniqueName: \"kubernetes.io/projected/89909b81-d1a4-4134-9692-9544cba7903b-kube-api-access-krfs6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:32.769467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.769439    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume" pod "a27b72ea-653a-47e3-9330-312df17c16cb" (UID: "a27b72ea-653a-47e3-9330-312df17c16cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:32.770930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.770903    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp" pod "a27b72ea-653a-47e3-9330-312df17c16cb" (UID: "a27b72ea-653a-47e3-9330-312df17c16cb"). InnerVolumeSpecName "kube-api-access-25cdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:32.870569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.870488    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b72ea-653a-47e3-9330-312df17c16cb-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:32.870569 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:32.870518    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25cdp\" (UniqueName: \"kubernetes.io/projected/a27b72ea-653a-47e3-9330-312df17c16cb-kube-api-access-25cdp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:33.544888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.544856    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"
Nov 04 00:33:33.545413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.544887    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7" event={"ID":"a27b72ea-653a-47e3-9330-312df17c16cb","Type":"ContainerDied","Data":"9c2798abee896b4aaec02f16c1e6e15aa627c9e5c7eef8451c62529220d699cb"}
Nov 04 00:33:33.545413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.544970    5074 scope.go:122] "RemoveContainer" containerID="8e0efaf924feaf67f7a049b4faf8105462bd94d96bfc62e2d7549f1063671a3f"
Nov 04 00:33:33.547649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.547625    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"
Nov 04 00:33:33.548295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.547624    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l" event={"ID":"89909b81-d1a4-4134-9692-9544cba7903b","Type":"ContainerDied","Data":"530781b24973cdf0380e8f60866a9efa808596e15baf115655dce76fe31ba158"}
Nov 04 00:33:33.558264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.558245    5074 scope.go:122] "RemoveContainer" containerID="de875d08255e379f7e1bd5214cce074e104d7e074563c1c0a6997a7c51a4d96c"
Nov 04 00:33:33.570131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.570096    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"]
Nov 04 00:33:33.572605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.572583    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5cb349-58d4f595f9-fn9t7"]
Nov 04 00:33:33.588827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.588798    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"]
Nov 04 00:33:33.592311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.592286    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5cb349-6598d94444-bfg2l"]
Nov 04 00:33:33.716459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.716428    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:33.782891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.782860    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca\") pod \"1e1c2495-9587-48c0-ba35-2be588219891\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") "
Nov 04 00:33:33.783098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.782947    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89\" (UniqueName: \"kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89\") pod \"1e1c2495-9587-48c0-ba35-2be588219891\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") "
Nov 04 00:33:33.783098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.783083    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca\") pod \"1e1c2495-9587-48c0-ba35-2be588219891\" (UID: \"1e1c2495-9587-48c0-ba35-2be588219891\") "
Nov 04 00:33:33.783318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.783291    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca" pod "1e1c2495-9587-48c0-ba35-2be588219891" (UID: "1e1c2495-9587-48c0-ba35-2be588219891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:33.783574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.783554    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:33.785541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.785516    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca" pod "1e1c2495-9587-48c0-ba35-2be588219891" (UID: "1e1c2495-9587-48c0-ba35-2be588219891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:33.785766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.785719    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89" pod "1e1c2495-9587-48c0-ba35-2be588219891" (UID: "1e1c2495-9587-48c0-ba35-2be588219891"). InnerVolumeSpecName "kube-api-access-pcs89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:33.885007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.884965    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcs89\" (UniqueName: \"kubernetes.io/projected/1e1c2495-9587-48c0-ba35-2be588219891-kube-api-access-pcs89\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:33.885007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:33.885006    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1c2495-9587-48c0-ba35-2be588219891-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:34.559859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:34.559821    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"
Nov 04 00:33:34.559859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:34.559833    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f" event={"ID":"1e1c2495-9587-48c0-ba35-2be588219891","Type":"ContainerDied","Data":"c8ff0c322214717e86c6a8ea4bf659f8bd698629ff0ebf46099797790fb88dad"}
Nov 04 00:33:34.560342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:34.559886    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ff0c322214717e86c6a8ea4bf659f8bd698629ff0ebf46099797790fb88dad"
Nov 04 00:33:35.019983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:35.019951    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89909b81-d1a4-4134-9692-9544cba7903b" path="/var/lib/kubelet/pods/89909b81-d1a4-4134-9692-9544cba7903b/volumes"
Nov 04 00:33:35.020424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:35.020409    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27b72ea-653a-47e3-9330-312df17c16cb" path="/var/lib/kubelet/pods/a27b72ea-653a-47e3-9330-312df17c16cb/volumes"
Nov 04 00:33:36.114876 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.114827    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" containerID="cri-o://1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b" gracePeriod=45
Nov 04 00:33:36.116204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.116166    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main" containerID="cri-o://653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba" gracePeriod=45
Nov 04 00:33:36.281215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.281164    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": dial tcp 10.132.0.100:8000: connect: connection refused"
Nov 04 00:33:36.427682 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.427653    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 04 00:33:36.455538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.455510    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 04 00:33:36.521598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521557    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home\") pod \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") "
Nov 04 00:33:36.521791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521614    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm\") pod \"292fcf1a-0854-403f-b440-9878ad3811cf\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") "
Nov 04 00:33:36.521791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521674    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs\") pod \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") "
Nov 04 00:33:36.521791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521709    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx\" (UniqueName: \"kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx\") pod \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") "
Nov 04 00:33:36.521791 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521772    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir\") pod \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") "
Nov 04 00:33:36.522067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521813    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn\" (UniqueName: \"kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn\") pod \"292fcf1a-0854-403f-b440-9878ad3811cf\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") "
Nov 04 00:33:36.522067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521846    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home\") pod \"292fcf1a-0854-403f-b440-9878ad3811cf\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") "
Nov 04 00:33:36.522067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521882    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache\") pod \"292fcf1a-0854-403f-b440-9878ad3811cf\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") "
Nov 04 00:33:36.522067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.521941    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs\") pod \"292fcf1a-0854-403f-b440-9878ad3811cf\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") "
Nov 04 00:33:36.522067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.522009    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache\") pod \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") "
Nov 04 00:33:36.522067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.522062    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir\") pod \"292fcf1a-0854-403f-b440-9878ad3811cf\" (UID: \"292fcf1a-0854-403f-b440-9878ad3811cf\") "
Nov 04 00:33:36.522392 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.522205    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm\") pod \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\" (UID: \"4f2a4304-fe1d-450b-88b6-f3547bc1c87b\") "
Nov 04 00:33:36.524357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.524319    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir" pod "4f2a4304-fe1d-450b-88b6-f3547bc1c87b" (UID: "4f2a4304-fe1d-450b-88b6-f3547bc1c87b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.524532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.524514    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home" pod "292fcf1a-0854-403f-b440-9878ad3811cf" (UID: "292fcf1a-0854-403f-b440-9878ad3811cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.524689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.524671    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache" pod "292fcf1a-0854-403f-b440-9878ad3811cf" (UID: "292fcf1a-0854-403f-b440-9878ad3811cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.524867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.524849    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache" pod "4f2a4304-fe1d-450b-88b6-f3547bc1c87b" (UID: "4f2a4304-fe1d-450b-88b6-f3547bc1c87b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.525023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.525006    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir" pod "292fcf1a-0854-403f-b440-9878ad3811cf" (UID: "292fcf1a-0854-403f-b440-9878ad3811cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.525491 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.525466    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home" pod "4f2a4304-fe1d-450b-88b6-f3547bc1c87b" (UID: "4f2a4304-fe1d-450b-88b6-f3547bc1c87b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.527561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.527526    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx" pod "4f2a4304-fe1d-450b-88b6-f3547bc1c87b" (UID: "4f2a4304-fe1d-450b-88b6-f3547bc1c87b"). InnerVolumeSpecName "kube-api-access-k6rwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:36.531386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.531348    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs" pod "292fcf1a-0854-403f-b440-9878ad3811cf" (UID: "292fcf1a-0854-403f-b440-9878ad3811cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:36.531519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.531383    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn" pod "292fcf1a-0854-403f-b440-9878ad3811cf" (UID: "292fcf1a-0854-403f-b440-9878ad3811cf"). InnerVolumeSpecName "kube-api-access-d85nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:36.531604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.531586    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm" pod "4f2a4304-fe1d-450b-88b6-f3547bc1c87b" (UID: "4f2a4304-fe1d-450b-88b6-f3547bc1c87b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.531694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.531670    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs" pod "4f2a4304-fe1d-450b-88b6-f3547bc1c87b" (UID: "4f2a4304-fe1d-450b-88b6-f3547bc1c87b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:36.532611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.532592    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm" pod "292fcf1a-0854-403f-b440-9878ad3811cf" (UID: "292fcf1a-0854-403f-b440-9878ad3811cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.579220 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.579181    5074 generic.go:356] "Generic (PLEG): container finished" podID="292fcf1a-0854-403f-b440-9878ad3811cf" containerID="1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b" exitCode=0
Nov 04 00:33:36.579380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.579255    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"
Nov 04 00:33:36.579380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.579265    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" event={"ID":"292fcf1a-0854-403f-b440-9878ad3811cf","Type":"ContainerDied","Data":"1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b"}
Nov 04 00:33:36.579380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.579310    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn" event={"ID":"292fcf1a-0854-403f-b440-9878ad3811cf","Type":"ContainerDied","Data":"5d3099c2cfe744855dfad65fda9e19dda66e0f089cf9da60835944ffae439fb7"}
Nov 04 00:33:36.579380 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.579331    5074 scope.go:122] "RemoveContainer" containerID="1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b"
Nov 04 00:33:36.581064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.581040    5074 generic.go:356] "Generic (PLEG): container finished" podID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerID="653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba" exitCode=0
Nov 04 00:33:36.581171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.581075    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" event={"ID":"4f2a4304-fe1d-450b-88b6-f3547bc1c87b","Type":"ContainerDied","Data":"653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba"}
Nov 04 00:33:36.581171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.581102    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"
Nov 04 00:33:36.581251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.581102    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs" event={"ID":"4f2a4304-fe1d-450b-88b6-f3547bc1c87b","Type":"ContainerDied","Data":"52fed4d5d777030cac964e0d2c062500afd2db53c642fd0683fb7036e1a7e176"}
Nov 04 00:33:36.592705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.591620    5074 scope.go:122] "RemoveContainer" containerID="1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b"
Nov 04 00:33:36.594054 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:36.593894    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b\": container with ID starting with 1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b not found: ID does not exist" containerID="1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b"
Nov 04 00:33:36.594054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.593939    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b"} err="failed to get container status \"1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b\": rpc error: code = NotFound desc = could not find container \"1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b\": container with ID starting with 1125568c43180f483f058c50438b1cdd6702f10d396b764fff18a358269d8a2b not found: ID does not exist"
Nov 04 00:33:36.594054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.593962    5074 scope.go:122] "RemoveContainer" containerID="653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba"
Nov 04 00:33:36.604431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.604405    5074 scope.go:122] "RemoveContainer" containerID="653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba"
Nov 04 00:33:36.604753 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:36.604713    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba\": container with ID starting with 653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba not found: ID does not exist" containerID="653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba"
Nov 04 00:33:36.604858 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.604758    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba"} err="failed to get container status \"653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba\": rpc error: code = NotFound desc = could not find container \"653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba\": container with ID starting with 653f6baf5354dda4f9d79e3632c6097be9274e580adbdc6885af79e528d941ba not found: ID does not exist"
Nov 04 00:33:36.612797 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.612774    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"]
Nov 04 00:33:36.618699 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.618674    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/premium-w1-5cb349-kserve-6f787c957-v5czs"]
Nov 04 00:33:36.623650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623623    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623647    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292fcf1a-0854-403f-b440-9878ad3811cf-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623657    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623666    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623674    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623682    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623689    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623696    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623705    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rwx\" (UniqueName: \"kubernetes.io/projected/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-kube-api-access-k6rwx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623713    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a4304-fe1d-450b-88b6-f3547bc1c87b-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623721    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d85nn\" (UniqueName: \"kubernetes.io/projected/292fcf1a-0854-403f-b440-9878ad3811cf-kube-api-access-d85nn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.623793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.623729    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292fcf1a-0854-403f-b440-9878ad3811cf-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.631657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.631599    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"]
Nov 04 00:33:36.635233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:36.635202    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-w1-5cb349-kserve-545c7887c6-6fpnn"]
Nov 04 00:33:37.019730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:37.019695    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" path="/var/lib/kubelet/pods/292fcf1a-0854-403f-b440-9878ad3811cf/volumes"
Nov 04 00:33:37.020267 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:37.020250    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" path="/var/lib/kubelet/pods/4f2a4304-fe1d-450b-88b6-f3547bc1c87b/volumes"
Nov 04 00:33:56.057764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:56.052009    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:33:56.104282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:33:56.104245    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:33:58.025947 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:33:58.025908    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:34:01.459335 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:01.459294    5074 scope.go:122] "RemoveContainer" containerID="ffe4585c828d00de75e79ede16eec857d83786827ba2142845f52ddb935dd568"
Nov 04 00:34:01.469679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:01.469648    5074 scope.go:122] "RemoveContainer" containerID="72d463c6d2aa85f030408a3dc15a8989ba50bebb4352f33abbd49ce3e97d91e9"
Nov 04 00:34:01.479268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:01.479246    5074 scope.go:122] "RemoveContainer" containerID="cd2b389895b9d31d806193bc59324255b583e4d31067e1cc5f63c82e7876ad57"
Nov 04 00:34:04.070762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:04.069895    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"]
Nov 04 00:34:04.070762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:04.070191    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy" containerID="cri-o://889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a" gracePeriod=30
Nov 04 00:34:05.954305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954227    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"]
Nov 04 00:34:05.954890 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954875    5074 cpu_manager.go:414] "containerMap: removing container" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main"
Nov 04 00:34:05.954951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954891    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main"
Nov 04 00:34:05.954951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954922    5074 cpu_manager.go:414] "containerMap: removing container" podUID="89909b81-d1a4-4134-9692-9544cba7903b" containerName="payload-processing"
Nov 04 00:34:05.954951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954929    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89909b81-d1a4-4134-9692-9544cba7903b" containerName="payload-processing"
Nov 04 00:34:05.954951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954945    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main"
Nov 04 00:34:05.954951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954950    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954964    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a27b72ea-653a-47e3-9330-312df17c16cb" containerName="payload-pre-processing"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954970    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27b72ea-653a-47e3-9330-312df17c16cb" containerName="payload-pre-processing"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954988    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1e1c2495-9587-48c0-ba35-2be588219891" containerName="revoke-keys"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.954993    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1c2495-9587-48c0-ba35-2be588219891" containerName="revoke-keys"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.955121    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27b72ea-653a-47e3-9330-312df17c16cb" containerName="payload-pre-processing"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.955140    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1c2495-9587-48c0-ba35-2be588219891" containerName="revoke-keys"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.955150    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="292fcf1a-0854-403f-b440-9878ad3811cf" containerName="main"
Nov 04 00:34:05.955160 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.955159    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="89909b81-d1a4-4134-9692-9544cba7903b" containerName="payload-processing"
Nov 04 00:34:05.955450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.955167    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2a4304-fe1d-450b-88b6-f3547bc1c87b" containerName="main"
Nov 04 00:34:05.960547 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.960515    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:05.965589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.965330    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:34:05.965846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.965762    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"]
Nov 04 00:34:05.967977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:05.967954    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:34:06.130577 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.130538    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.130790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.130672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.130790 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.130709    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992w9\" (UniqueName: \"kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.232081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.232001    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.232081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.232050    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.232311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.232103    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992w9\" (UniqueName: \"kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.232941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.232911    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.234891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.234862    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.245084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.245058    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992w9\" (UniqueName: \"kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9\") pod \"maas-api-revoke-sub-6f141690-mm7kl\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.290146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.290108    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:06.575919 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.575890    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"]
Nov 04 00:34:06.577140 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:34:06.577110    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8780bcf_f7bd_428c_aa5e_94faf82ae061.slice/crio-54ec0649e38fe92f97ecb6d2fef3eea899c53675c17381f131369c2004889994 WatchSource:0}: Error finding container 54ec0649e38fe92f97ecb6d2fef3eea899c53675c17381f131369c2004889994: Status 404 returned error can't find the container with id 54ec0649e38fe92f97ecb6d2fef3eea899c53675c17381f131369c2004889994
Nov 04 00:34:06.831815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:06.831329    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl" event={"ID":"f8780bcf-f7bd-428c-aa5e-94faf82ae061","Type":"ContainerStarted","Data":"54ec0649e38fe92f97ecb6d2fef3eea899c53675c17381f131369c2004889994"}
Nov 04 00:34:07.843843 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:07.843807    5074 generic.go:356] "Generic (PLEG): container finished" podID="f8780bcf-f7bd-428c-aa5e-94faf82ae061" containerID="87cc59d8b5e8b495b965f9dc3dc6bfde59a47fe89512d4b2bd9b5b737dae6bad" exitCode=0
Nov 04 00:34:07.844305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:07.843901    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl" event={"ID":"f8780bcf-f7bd-428c-aa5e-94faf82ae061","Type":"ContainerDied","Data":"87cc59d8b5e8b495b965f9dc3dc6bfde59a47fe89512d4b2bd9b5b737dae6bad"}
Nov 04 00:34:09.014923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.014894    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:09.168761 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.168653    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9\" (UniqueName: \"kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9\") pod \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") "
Nov 04 00:34:09.168945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.168828    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca\") pod \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") "
Nov 04 00:34:09.168945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.168917    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca\") pod \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\" (UID: \"f8780bcf-f7bd-428c-aa5e-94faf82ae061\") "
Nov 04 00:34:09.169297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.169267    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca" pod "f8780bcf-f7bd-428c-aa5e-94faf82ae061" (UID: "f8780bcf-f7bd-428c-aa5e-94faf82ae061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:34:09.169439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.169426    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:09.171000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.170971    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9" pod "f8780bcf-f7bd-428c-aa5e-94faf82ae061" (UID: "f8780bcf-f7bd-428c-aa5e-94faf82ae061"). InnerVolumeSpecName "kube-api-access-992w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:09.171134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.171064    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca" pod "f8780bcf-f7bd-428c-aa5e-94faf82ae061" (UID: "f8780bcf-f7bd-428c-aa5e-94faf82ae061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:34:09.270338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.270303    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-992w9\" (UniqueName: \"kubernetes.io/projected/f8780bcf-f7bd-428c-aa5e-94faf82ae061-kube-api-access-992w9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:09.270338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.270333    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8780bcf-f7bd-428c-aa5e-94faf82ae061-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:09.861653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.861615    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"
Nov 04 00:34:09.861885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.861620    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl" event={"ID":"f8780bcf-f7bd-428c-aa5e-94faf82ae061","Type":"ContainerDied","Data":"54ec0649e38fe92f97ecb6d2fef3eea899c53675c17381f131369c2004889994"}
Nov 04 00:34:09.861885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:09.861823    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54ec0649e38fe92f97ecb6d2fef3eea899c53675c17381f131369c2004889994"
Nov 04 00:34:10.359908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.359875    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 04 00:34:10.483355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483322    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483376    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483411    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483531    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483568    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483614    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483648    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.483734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483659    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:10.483734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483701    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.484057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483775    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket\") pod \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\" (UID: \"f16f47b3-31ad-4e27-bdc8-aa26c334ab54\") "
Nov 04 00:34:10.484057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483832    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:10.484057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.483874    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:10.484225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484056    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:10.484225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484156    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:34:10.484467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484445    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.484557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484473    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.484557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484487    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.484557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484500    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.484557 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.484514    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.486043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.486018    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "kube-api-access-vrj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:10.486164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.486081    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:10.486415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.486389    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:10.486536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.486402    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo" pod "f16f47b3-31ad-4e27-bdc8-aa26c334ab54" (UID: "f16f47b3-31ad-4e27-bdc8-aa26c334ab54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:34:10.585928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.585892    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.585928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.585923    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.585928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.585935    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrj7b\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-kube-api-access-vrj7b\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.586180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.585947    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f47b3-31ad-4e27-bdc8-aa26c334ab54-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:10.870316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.870223    5074 generic.go:356] "Generic (PLEG): container finished" podID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerID="889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a" exitCode=0
Nov 04 00:34:10.870316 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.870290    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"
Nov 04 00:34:10.870523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.870290    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" event={"ID":"f16f47b3-31ad-4e27-bdc8-aa26c334ab54","Type":"ContainerDied","Data":"889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a"}
Nov 04 00:34:10.870523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.870419    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5" event={"ID":"f16f47b3-31ad-4e27-bdc8-aa26c334ab54","Type":"ContainerDied","Data":"fc4d5a337b708189173876dc9a2e1ba71abc9c08f42d4214912be49d390a86df"}
Nov 04 00:34:10.870523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.870444    5074 scope.go:122] "RemoveContainer" containerID="889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a"
Nov 04 00:34:10.882490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.882471    5074 scope.go:122] "RemoveContainer" containerID="889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a"
Nov 04 00:34:10.882802 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:34:10.882777    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a\": container with ID starting with 889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a not found: ID does not exist" containerID="889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a"
Nov 04 00:34:10.882866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.882811    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a"} err="failed to get container status \"889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a\": rpc error: code = NotFound desc = could not find container \"889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a\": container with ID starting with 889f26d99e691cc508ca69cca096cd7fc24ff653684f32d1f3e1ec49502a462a not found: ID does not exist"
Nov 04 00:34:10.898338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.898314    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"]
Nov 04 00:34:10.903427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:10.903401    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c67j5"]
Nov 04 00:34:11.022581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:11.022546    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" path="/var/lib/kubelet/pods/f16f47b3-31ad-4e27-bdc8-aa26c334ab54/volumes"
Nov 04 00:34:16.161437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.161391    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"]
Nov 04 00:34:16.162034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.162018    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f8780bcf-f7bd-428c-aa5e-94faf82ae061" containerName="revoke-keys"
Nov 04 00:34:16.162105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.162034    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8780bcf-f7bd-428c-aa5e-94faf82ae061" containerName="revoke-keys"
Nov 04 00:34:16.162105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.162058    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy"
Nov 04 00:34:16.162105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.162065    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy"
Nov 04 00:34:16.162208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.162181    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8780bcf-f7bd-428c-aa5e-94faf82ae061" containerName="revoke-keys"
Nov 04 00:34:16.162208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.162195    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16f47b3-31ad-4e27-bdc8-aa26c334ab54" containerName="istio-proxy"
Nov 04 00:34:16.167241 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.167225    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.173429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.173398    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"]
Nov 04 00:34:16.174359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.174339    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:34:16.174450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.174431    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:34:16.255083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.255046    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vtpq\" (UniqueName: \"kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.255287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.255106    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.255356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.255288    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.355920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.355876    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vtpq\" (UniqueName: \"kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.356141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.355932    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.356141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.356073    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.356676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.356650    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.358495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.358470    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.369053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.369030    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vtpq\" (UniqueName: \"kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq\") pod \"maas-api-revoke-sub-3bf83ed2-2wzw8\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.483496 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.483463    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:16.774624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.774592    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"]
Nov 04 00:34:16.775384 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:34:16.775360    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda920b9a8_f55f_45b7_9468_02a1ed04a7b3.slice/crio-7ec983915435c5203b6ebd0e98dac3b1321f622bedf147fc1ce64ebceb241573 WatchSource:0}: Error finding container 7ec983915435c5203b6ebd0e98dac3b1321f622bedf147fc1ce64ebceb241573: Status 404 returned error can't find the container with id 7ec983915435c5203b6ebd0e98dac3b1321f622bedf147fc1ce64ebceb241573
Nov 04 00:34:16.916929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:16.916889    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8" event={"ID":"a920b9a8-f55f-45b7-9468-02a1ed04a7b3","Type":"ContainerStarted","Data":"7ec983915435c5203b6ebd0e98dac3b1321f622bedf147fc1ce64ebceb241573"}
Nov 04 00:34:17.926079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:17.925980    5074 generic.go:356] "Generic (PLEG): container finished" podID="a920b9a8-f55f-45b7-9468-02a1ed04a7b3" containerID="e1a53cca36d476a268a6ac7aa3287c2a472a205acb2e7e4addc0d3989719c3a0" exitCode=0
Nov 04 00:34:17.926079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:17.926068    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8" event={"ID":"a920b9a8-f55f-45b7-9468-02a1ed04a7b3","Type":"ContainerDied","Data":"e1a53cca36d476a268a6ac7aa3287c2a472a205acb2e7e4addc0d3989719c3a0"}
Nov 04 00:34:19.099498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.099464    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:19.189604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.189509    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca\") pod \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") "
Nov 04 00:34:19.189604 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.189572    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq\" (UniqueName: \"kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq\") pod \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") "
Nov 04 00:34:19.189889 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.189852    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca\") pod \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\" (UID: \"a920b9a8-f55f-45b7-9468-02a1ed04a7b3\") "
Nov 04 00:34:19.189960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.189903    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca" pod "a920b9a8-f55f-45b7-9468-02a1ed04a7b3" (UID: "a920b9a8-f55f-45b7-9468-02a1ed04a7b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:34:19.190288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.190262    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:19.191826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.191731    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq" pod "a920b9a8-f55f-45b7-9468-02a1ed04a7b3" (UID: "a920b9a8-f55f-45b7-9468-02a1ed04a7b3"). InnerVolumeSpecName "kube-api-access-8vtpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:19.191988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.191969    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca" pod "a920b9a8-f55f-45b7-9468-02a1ed04a7b3" (UID: "a920b9a8-f55f-45b7-9468-02a1ed04a7b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:34:19.291713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.291673    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:19.291713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.291704    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vtpq\" (UniqueName: \"kubernetes.io/projected/a920b9a8-f55f-45b7-9468-02a1ed04a7b3-kube-api-access-8vtpq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:19.943289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.943255    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"
Nov 04 00:34:19.943478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.943287    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8" event={"ID":"a920b9a8-f55f-45b7-9468-02a1ed04a7b3","Type":"ContainerDied","Data":"7ec983915435c5203b6ebd0e98dac3b1321f622bedf147fc1ce64ebceb241573"}
Nov 04 00:34:19.943478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:19.943371    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec983915435c5203b6ebd0e98dac3b1321f622bedf147fc1ce64ebceb241573"
Nov 04 00:34:26.813785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:26.811398    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"]
Nov 04 00:34:28.025448 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:34:28.025399    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:34:28.543187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.543146    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"]
Nov 04 00:34:28.544153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.544132    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a920b9a8-f55f-45b7-9468-02a1ed04a7b3" containerName="revoke-keys"
Nov 04 00:34:28.544208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.544157    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a920b9a8-f55f-45b7-9468-02a1ed04a7b3" containerName="revoke-keys"
Nov 04 00:34:28.544302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.544291    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a920b9a8-f55f-45b7-9468-02a1ed04a7b3" containerName="revoke-keys"
Nov 04 00:34:28.548956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.548933    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.556246 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.555914    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8093f5-openshift-default-dockercfg-fkkkn\""
Nov 04 00:34:28.557813 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.557790    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"]
Nov 04 00:34:28.601753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.601718    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.601920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.601844    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.601996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.601969    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.602101 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.602081    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.602146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.602117    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.602197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.602178    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.602237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.602221    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.602294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.602283    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.602344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.602331    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmfs\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.703835 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.703794    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.703905    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.703984    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704054    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704081    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704118    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704144    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704199    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704264    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmfs\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704323    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704613    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.704936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.704893    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.705117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.705094    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.705323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.705297    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.706538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.706516    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.707401 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.707376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.717136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.717110    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.717701 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.717673    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmfs\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:28.867195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:28.867117    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:29.160153 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:34:29.160117    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b34d86_f828_4b79_ac76_cac5c747ec69.slice/crio-acc83446fae10155347c6199824f30f94f04739fb90cb384d53376e6f2252298 WatchSource:0}: Error finding container acc83446fae10155347c6199824f30f94f04739fb90cb384d53376e6f2252298: Status 404 returned error can't find the container with id acc83446fae10155347c6199824f30f94f04739fb90cb384d53376e6f2252298
Nov 04 00:34:29.160502 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:29.160287    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"]
Nov 04 00:34:29.163086 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:29.163049    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:34:29.163190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:29.163132    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:34:29.163190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:29.163178    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:34:30.038955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.035785    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"]
Nov 04 00:34:30.042579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.042534    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2a1248-jnn5s"]
Nov 04 00:34:30.046750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.046698    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" event={"ID":"89b34d86-f828-4b79-ac76-cac5c747ec69","Type":"ContainerStarted","Data":"caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060"}
Nov 04 00:34:30.046873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.046804    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" event={"ID":"89b34d86-f828-4b79-ac76-cac5c747ec69","Type":"ContainerStarted","Data":"acc83446fae10155347c6199824f30f94f04739fb90cb384d53376e6f2252298"}
Nov 04 00:34:30.073277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.073231    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podStartSLOduration=2.073217891 podStartE2EDuration="2.073217891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:34:30.070200978 +0000 UTC m=+3935.169801148" watchObservedRunningTime="2026-11-04 00:34:30.073217891 +0000 UTC m=+3935.172818034"
Nov 04 00:34:30.867563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.867524    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:30.869796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.869734    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 04 00:34:30.870018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:30.869986    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 04 00:34:31.043869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:31.043829    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbd11a0-78cb-4572-a6f7-d3010494bbc6" path="/var/lib/kubelet/pods/adbd11a0-78cb-4572-a6f7-d3010494bbc6/volumes"
Nov 04 00:34:31.868407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:31.868367    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 04 00:34:31.868864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:31.868433    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 04 00:34:32.049073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:32.049039    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"]
Nov 04 00:34:32.867811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:32.867772    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 04 00:34:32.868028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:32.867829    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 04 00:34:33.868165 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:33.868121    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 04 00:34:33.868662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:33.868187    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 04 00:34:34.868375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:34.868336    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 04 00:34:34.868888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:34.868389    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 04 00:34:35.872099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:35.872061    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:35.872566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:35.872551    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:35.873388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:35.873362    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:34:36.398173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.398127    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"]
Nov 04 00:34:36.402842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.402812    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.409134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.409105    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:34:36.409342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.409155    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:34:36.411793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.411769    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"]
Nov 04 00:34:36.553378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.553339    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.553571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.553514    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.553626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.553592    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhj9\" (UniqueName: \"kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.655163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.655060    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.655337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.655167    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.655337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.655207    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhj9\" (UniqueName: \"kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.655942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.655914    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.657753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.657713    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.668369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.668340    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhj9\" (UniqueName: \"kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9\") pod \"maas-api-revoke-sub-efc98973-697d6\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:36.723459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:36.723421    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:37.011007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:37.010975    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"]
Nov 04 00:34:37.012504 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:34:37.012476    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3762366b_4fdb_40ad_ab98_4241d08dc3cc.slice/crio-230584a17ef037dac3dff0ddc774980aaa2d18cee907c6ceb17c155b76f1553a WatchSource:0}: Error finding container 230584a17ef037dac3dff0ddc774980aaa2d18cee907c6ceb17c155b76f1553a: Status 404 returned error can't find the container with id 230584a17ef037dac3dff0ddc774980aaa2d18cee907c6ceb17c155b76f1553a
Nov 04 00:34:37.118081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:37.118043    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6" event={"ID":"3762366b-4fdb-40ad-ab98-4241d08dc3cc","Type":"ContainerStarted","Data":"230584a17ef037dac3dff0ddc774980aaa2d18cee907c6ceb17c155b76f1553a"}
Nov 04 00:34:38.127236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:38.127148    5074 generic.go:356] "Generic (PLEG): container finished" podID="3762366b-4fdb-40ad-ab98-4241d08dc3cc" containerID="ed392eb6d5224e1e3569751eb4fb0c1c175900c033ad9a5db1b73fe6cf6d423c" exitCode=0
Nov 04 00:34:38.127964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:38.127346    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6" event={"ID":"3762366b-4fdb-40ad-ab98-4241d08dc3cc","Type":"ContainerDied","Data":"ed392eb6d5224e1e3569751eb4fb0c1c175900c033ad9a5db1b73fe6cf6d423c"}
Nov 04 00:34:38.867295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:38.867257    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"]
Nov 04 00:34:38.878221 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:38.878185    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"]
Nov 04 00:34:39.323072 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.323039    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:39.492334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.492301    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca\") pod \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") "
Nov 04 00:34:39.492518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.492354    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9\" (UniqueName: \"kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9\") pod \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") "
Nov 04 00:34:39.492518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.492452    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca\") pod \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\" (UID: \"3762366b-4fdb-40ad-ab98-4241d08dc3cc\") "
Nov 04 00:34:39.492838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.492814    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca" pod "3762366b-4fdb-40ad-ab98-4241d08dc3cc" (UID: "3762366b-4fdb-40ad-ab98-4241d08dc3cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:34:39.494723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.494696    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9" pod "3762366b-4fdb-40ad-ab98-4241d08dc3cc" (UID: "3762366b-4fdb-40ad-ab98-4241d08dc3cc"). InnerVolumeSpecName "kube-api-access-khhj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:39.494851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.494803    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca" pod "3762366b-4fdb-40ad-ab98-4241d08dc3cc" (UID: "3762366b-4fdb-40ad-ab98-4241d08dc3cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:34:39.609370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.609328    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:39.609789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.609707    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhj9\" (UniqueName: \"kubernetes.io/projected/3762366b-4fdb-40ad-ab98-4241d08dc3cc-kube-api-access-khhj9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:39.609789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:39.609766    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3762366b-4fdb-40ad-ab98-4241d08dc3cc-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:40.154909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:40.154874    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"
Nov 04 00:34:40.155113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:40.154907    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6" event={"ID":"3762366b-4fdb-40ad-ab98-4241d08dc3cc","Type":"ContainerDied","Data":"230584a17ef037dac3dff0ddc774980aaa2d18cee907c6ceb17c155b76f1553a"}
Nov 04 00:34:40.155113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:40.154958    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230584a17ef037dac3dff0ddc774980aaa2d18cee907c6ceb17c155b76f1553a"
Nov 04 00:34:41.847469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:41.847421    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj" podUID="33318433-8255-441e-b24a-9aa8d1abca1e" containerName="main" containerID="cri-o://c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25" gracePeriod=45
Nov 04 00:34:42.129617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.129592    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:34:42.148596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148570    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs\") pod \"33318433-8255-441e-b24a-9aa8d1abca1e\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") "
Nov 04 00:34:42.148725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148704    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir\") pod \"33318433-8255-441e-b24a-9aa8d1abca1e\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") "
Nov 04 00:34:42.148847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148831    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache\") pod \"33318433-8255-441e-b24a-9aa8d1abca1e\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") "
Nov 04 00:34:42.148918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148900    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz\" (UniqueName: \"kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz\") pod \"33318433-8255-441e-b24a-9aa8d1abca1e\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") "
Nov 04 00:34:42.148987 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148941    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home\") pod \"33318433-8255-441e-b24a-9aa8d1abca1e\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") "
Nov 04 00:34:42.148987 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148958    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir" pod "33318433-8255-441e-b24a-9aa8d1abca1e" (UID: "33318433-8255-441e-b24a-9aa8d1abca1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:42.149100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.148985    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm\") pod \"33318433-8255-441e-b24a-9aa8d1abca1e\" (UID: \"33318433-8255-441e-b24a-9aa8d1abca1e\") "
Nov 04 00:34:42.149100 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.149083    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache" pod "33318433-8255-441e-b24a-9aa8d1abca1e" (UID: "33318433-8255-441e-b24a-9aa8d1abca1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:42.149254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.149229    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home" pod "33318433-8255-441e-b24a-9aa8d1abca1e" (UID: "33318433-8255-441e-b24a-9aa8d1abca1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:42.149445 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.149428    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:42.149551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.149451    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:42.149551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.149480    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:42.151582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.151558    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm" pod "33318433-8255-441e-b24a-9aa8d1abca1e" (UID: "33318433-8255-441e-b24a-9aa8d1abca1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:42.151697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.151682    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz" pod "33318433-8255-441e-b24a-9aa8d1abca1e" (UID: "33318433-8255-441e-b24a-9aa8d1abca1e"). InnerVolumeSpecName "kube-api-access-z4xzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:42.186263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.186228    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs" pod "33318433-8255-441e-b24a-9aa8d1abca1e" (UID: "33318433-8255-441e-b24a-9aa8d1abca1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:34:42.194971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.194937    5074 generic.go:356] "Generic (PLEG): container finished" podID="33318433-8255-441e-b24a-9aa8d1abca1e" containerID="c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25" exitCode=0
Nov 04 00:34:42.195368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.195002    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"
Nov 04 00:34:42.195368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.195038    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj" event={"ID":"33318433-8255-441e-b24a-9aa8d1abca1e","Type":"ContainerDied","Data":"c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25"}
Nov 04 00:34:42.195368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.195077    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj" event={"ID":"33318433-8255-441e-b24a-9aa8d1abca1e","Type":"ContainerDied","Data":"acf96e0f77a288b364a56eb55421c28c0c32325ae77479c70e46786337693485"}
Nov 04 00:34:42.195368 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.195116    5074 scope.go:122] "RemoveContainer" containerID="c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25"
Nov 04 00:34:42.209830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.209809    5074 scope.go:122] "RemoveContainer" containerID="c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25"
Nov 04 00:34:42.210171 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:34:42.210147    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25\": container with ID starting with c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25 not found: ID does not exist" containerID="c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25"
Nov 04 00:34:42.210251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.210181    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25"} err="failed to get container status \"c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25\": rpc error: code = NotFound desc = could not find container \"c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25\": container with ID starting with c9d080fa434d8e94fe16e9d3e794d71d94f3e917b8f7371a5a67bf8f17bdfd25 not found: ID does not exist"
Nov 04 00:34:42.226598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.226574    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"]
Nov 04 00:34:42.230041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.230013    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-58b75d/rate-test-model-58b75d-kserve-5689484845-9pgrj"]
Nov 04 00:34:42.251183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.251155    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xzz\" (UniqueName: \"kubernetes.io/projected/33318433-8255-441e-b24a-9aa8d1abca1e-kube-api-access-z4xzz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:42.251183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.251182    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33318433-8255-441e-b24a-9aa8d1abca1e-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:42.251330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:42.251192    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33318433-8255-441e-b24a-9aa8d1abca1e-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:43.022064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:43.022026    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33318433-8255-441e-b24a-9aa8d1abca1e" path="/var/lib/kubelet/pods/33318433-8255-441e-b24a-9aa8d1abca1e/volumes"
Nov 04 00:34:43.868116 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:43.868069    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" podUID="ad917029-10ba-40c9-9b87-988c39079a64" containerName="payload-processing" containerID="cri-o://f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e" gracePeriod=25
Nov 04 00:34:43.878779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:43.878731    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" podUID="36656517-378e-4358-b97f-1fd76762cf94" containerName="payload-pre-processing" containerID="cri-o://ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0" gracePeriod=25
Nov 04 00:34:46.685035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.684997    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"]
Nov 04 00:34:46.685609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.685595    5074 cpu_manager.go:414] "containerMap: removing container" podUID="3762366b-4fdb-40ad-ab98-4241d08dc3cc" containerName="revoke-keys"
Nov 04 00:34:46.685658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.685611    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3762366b-4fdb-40ad-ab98-4241d08dc3cc" containerName="revoke-keys"
Nov 04 00:34:46.685658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.685632    5074 cpu_manager.go:414] "containerMap: removing container" podUID="33318433-8255-441e-b24a-9aa8d1abca1e" containerName="main"
Nov 04 00:34:46.685658 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.685639    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33318433-8255-441e-b24a-9aa8d1abca1e" containerName="main"
Nov 04 00:34:46.685819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.685772    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3762366b-4fdb-40ad-ab98-4241d08dc3cc" containerName="revoke-keys"
Nov 04 00:34:46.685819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.685783    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="33318433-8255-441e-b24a-9aa8d1abca1e" containerName="main"
Nov 04 00:34:46.690869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.690847    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.697538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.697515    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"]
Nov 04 00:34:46.729819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.729785    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:34:46.729985 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.729838    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:34:46.811904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.811864    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.812117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.812005    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.812117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.812093    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkszh\" (UniqueName: \"kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.912719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.912681    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.912908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.912796    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkszh\" (UniqueName: \"kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.912908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.912901    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.913399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.913379    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.915201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.915177    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:46.925675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:46.925648    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkszh\" (UniqueName: \"kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh\") pod \"maas-api-revoke-sub-f09b1806-5bql5\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:47.006887 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.006798    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:47.159929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.159886    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh" podUID="2e436f23-780c-4727-8e27-9120f9d533f8" containerName="main" containerID="cri-o://8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44" gracePeriod=45
Nov 04 00:34:47.297350 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:34:47.297310    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce218f33_c0f0_428e_9f5b_e30bbe253b61.slice/crio-4ec76583492e1d5c177ab75480893471814a6b5b6902130a758e652ad706f754 WatchSource:0}: Error finding container 4ec76583492e1d5c177ab75480893471814a6b5b6902130a758e652ad706f754: Status 404 returned error can't find the container with id 4ec76583492e1d5c177ab75480893471814a6b5b6902130a758e652ad706f754
Nov 04 00:34:47.298669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.298641    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"]
Nov 04 00:34:47.431230 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.431201    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:34:47.521035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521006    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir\") pod \"2e436f23-780c-4727-8e27-9120f9d533f8\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") "
Nov 04 00:34:47.521223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521116    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm\") pod \"2e436f23-780c-4727-8e27-9120f9d533f8\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") "
Nov 04 00:34:47.521223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521151    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache\") pod \"2e436f23-780c-4727-8e27-9120f9d533f8\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") "
Nov 04 00:34:47.521223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521182    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home\") pod \"2e436f23-780c-4727-8e27-9120f9d533f8\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") "
Nov 04 00:34:47.521223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521204    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s\" (UniqueName: \"kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s\") pod \"2e436f23-780c-4727-8e27-9120f9d533f8\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") "
Nov 04 00:34:47.521475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521246    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs\") pod \"2e436f23-780c-4727-8e27-9120f9d533f8\" (UID: \"2e436f23-780c-4727-8e27-9120f9d533f8\") "
Nov 04 00:34:47.521475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521315    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir" pod "2e436f23-780c-4727-8e27-9120f9d533f8" (UID: "2e436f23-780c-4727-8e27-9120f9d533f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:47.521475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521436    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache" pod "2e436f23-780c-4727-8e27-9120f9d533f8" (UID: "2e436f23-780c-4727-8e27-9120f9d533f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:47.521475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521454    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home" pod "2e436f23-780c-4727-8e27-9120f9d533f8" (UID: "2e436f23-780c-4727-8e27-9120f9d533f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:47.521918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521890    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:47.521918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521916    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:47.522129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.521930    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:47.524027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.524001    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs" pod "2e436f23-780c-4727-8e27-9120f9d533f8" (UID: "2e436f23-780c-4727-8e27-9120f9d533f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:34:47.524027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.523988    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm" pod "2e436f23-780c-4727-8e27-9120f9d533f8" (UID: "2e436f23-780c-4727-8e27-9120f9d533f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:34:47.524208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.524186    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s" pod "2e436f23-780c-4727-8e27-9120f9d533f8" (UID: "2e436f23-780c-4727-8e27-9120f9d533f8"). InnerVolumeSpecName "kube-api-access-lkm8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:47.622969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.622936    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e436f23-780c-4727-8e27-9120f9d533f8-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:47.622969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.622966    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkm8s\" (UniqueName: \"kubernetes.io/projected/2e436f23-780c-4727-8e27-9120f9d533f8-kube-api-access-lkm8s\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:47.622969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:47.622976    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e436f23-780c-4727-8e27-9120f9d533f8-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:48.270208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.270173    5074 generic.go:356] "Generic (PLEG): container finished" podID="ce218f33-c0f0-428e-9f5b-e30bbe253b61" containerID="d39ee780162ad0fc497f24463009aa39b6660cbf0d8d0edb6061ae1f625a6bf0" exitCode=0
Nov 04 00:34:48.270657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.270257    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5" event={"ID":"ce218f33-c0f0-428e-9f5b-e30bbe253b61","Type":"ContainerDied","Data":"d39ee780162ad0fc497f24463009aa39b6660cbf0d8d0edb6061ae1f625a6bf0"}
Nov 04 00:34:48.270657 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.270324    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5" event={"ID":"ce218f33-c0f0-428e-9f5b-e30bbe253b61","Type":"ContainerStarted","Data":"4ec76583492e1d5c177ab75480893471814a6b5b6902130a758e652ad706f754"}
Nov 04 00:34:48.271729 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.271707    5074 generic.go:356] "Generic (PLEG): container finished" podID="2e436f23-780c-4727-8e27-9120f9d533f8" containerID="8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44" exitCode=0
Nov 04 00:34:48.271859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.271758    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"
Nov 04 00:34:48.271859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.271782    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh" event={"ID":"2e436f23-780c-4727-8e27-9120f9d533f8","Type":"ContainerDied","Data":"8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44"}
Nov 04 00:34:48.271859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.271833    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh" event={"ID":"2e436f23-780c-4727-8e27-9120f9d533f8","Type":"ContainerDied","Data":"dbe4a68b58b068f68cbcd7be5a9644fd53c49bfd4e567100fbb15d848fabe74f"}
Nov 04 00:34:48.271859 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.271854    5074 scope.go:122] "RemoveContainer" containerID="8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44"
Nov 04 00:34:48.282415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.282399    5074 scope.go:122] "RemoveContainer" containerID="8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44"
Nov 04 00:34:48.282706 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:34:48.282686    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44\": container with ID starting with 8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44 not found: ID does not exist" containerID="8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44"
Nov 04 00:34:48.282812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.282715    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44"} err="failed to get container status \"8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44\": rpc error: code = NotFound desc = could not find container \"8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44\": container with ID starting with 8facbace5cb46fbe52ead2f0ad7395de1f412a56f681ca82fd3ec8e1cbbe6d44 not found: ID does not exist"
Nov 04 00:34:48.308750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.308711    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"]
Nov 04 00:34:48.323990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:48.323953    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8093f5/rate-test-model-8093f5-kserve-7c9cf68557-rwtfh"]
Nov 04 00:34:49.020240 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.020202    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e436f23-780c-4727-8e27-9120f9d533f8" path="/var/lib/kubelet/pods/2e436f23-780c-4727-8e27-9120f9d533f8/volumes"
Nov 04 00:34:49.449948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.449921    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:49.549488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.549413    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca\") pod \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") "
Nov 04 00:34:49.549639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.549599    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh\" (UniqueName: \"kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh\") pod \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") "
Nov 04 00:34:49.549639 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.549635    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca\") pod \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\" (UID: \"ce218f33-c0f0-428e-9f5b-e30bbe253b61\") "
Nov 04 00:34:49.550072 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.550045    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca" pod "ce218f33-c0f0-428e-9f5b-e30bbe253b61" (UID: "ce218f33-c0f0-428e-9f5b-e30bbe253b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:34:49.551542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.551515    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca" pod "ce218f33-c0f0-428e-9f5b-e30bbe253b61" (UID: "ce218f33-c0f0-428e-9f5b-e30bbe253b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:34:49.551637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.551621    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh" pod "ce218f33-c0f0-428e-9f5b-e30bbe253b61" (UID: "ce218f33-c0f0-428e-9f5b-e30bbe253b61"). InnerVolumeSpecName "kube-api-access-lkszh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:34:49.651106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.651065    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkszh\" (UniqueName: \"kubernetes.io/projected/ce218f33-c0f0-428e-9f5b-e30bbe253b61-kube-api-access-lkszh\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:49.651106 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.651100    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:49.651303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:49.651115    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce218f33-c0f0-428e-9f5b-e30bbe253b61-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:34:50.070634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:50.070581    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" podUID="36656517-378e-4358-b97f-1fd76762cf94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:34:50.070860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:50.070657    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" podUID="ad917029-10ba-40c9-9b87-988c39079a64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:34:50.290708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:50.290675    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"
Nov 04 00:34:50.290906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:50.290707    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5" event={"ID":"ce218f33-c0f0-428e-9f5b-e30bbe253b61","Type":"ContainerDied","Data":"4ec76583492e1d5c177ab75480893471814a6b5b6902130a758e652ad706f754"}
Nov 04 00:34:50.290906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:50.290775    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ec76583492e1d5c177ab75480893471814a6b5b6902130a758e652ad706f754"
Nov 04 00:34:57.406537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:34:57.406501    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"]
Nov 04 00:34:58.023459 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:34:58.023417    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:35:00.070589 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:00.070550    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" podUID="36656517-378e-4358-b97f-1fd76762cf94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:00.071041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:00.070566    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" podUID="ad917029-10ba-40c9-9b87-988c39079a64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:02.117185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:02.117145    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"]
Nov 04 00:35:04.184275 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.184249    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:35:04.199856 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.199831    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:35:04.239411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.239379    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume\") pod \"ad917029-10ba-40c9-9b87-988c39079a64\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") "
Nov 04 00:35:04.239612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.239466    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr\" (UniqueName: \"kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr\") pod \"36656517-378e-4358-b97f-1fd76762cf94\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") "
Nov 04 00:35:04.239612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.239504    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume\") pod \"36656517-378e-4358-b97f-1fd76762cf94\" (UID: \"36656517-378e-4358-b97f-1fd76762cf94\") "
Nov 04 00:35:04.240034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.240010    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume" pod "ad917029-10ba-40c9-9b87-988c39079a64" (UID: "ad917029-10ba-40c9-9b87-988c39079a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:04.240173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.240039    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume" pod "36656517-378e-4358-b97f-1fd76762cf94" (UID: "36656517-378e-4358-b97f-1fd76762cf94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:04.240266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.240222    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22\" (UniqueName: \"kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22\") pod \"ad917029-10ba-40c9-9b87-988c39079a64\" (UID: \"ad917029-10ba-40c9-9b87-988c39079a64\") "
Nov 04 00:35:04.240695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.240676    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad917029-10ba-40c9-9b87-988c39079a64-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:04.240824 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.240699    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36656517-378e-4358-b97f-1fd76762cf94-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:04.242412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.242387    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr" pod "36656517-378e-4358-b97f-1fd76762cf94" (UID: "36656517-378e-4358-b97f-1fd76762cf94"). InnerVolumeSpecName "kube-api-access-5wfpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:04.242914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.242895    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22" pod "ad917029-10ba-40c9-9b87-988c39079a64" (UID: "ad917029-10ba-40c9-9b87-988c39079a64"). InnerVolumeSpecName "kube-api-access-z8t22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:04.341379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.341344    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfpr\" (UniqueName: \"kubernetes.io/projected/36656517-378e-4358-b97f-1fd76762cf94-kube-api-access-5wfpr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:04.341379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.341378    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8t22\" (UniqueName: \"kubernetes.io/projected/ad917029-10ba-40c9-9b87-988c39079a64-kube-api-access-z8t22\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.402048    5074 generic.go:356] "Generic (PLEG): container finished" podID="ad917029-10ba-40c9-9b87-988c39079a64" containerID="f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e" exitCode=0
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.402100    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.402129    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" event={"ID":"ad917029-10ba-40c9-9b87-988c39079a64","Type":"ContainerDied","Data":"f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e"}
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.402191    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt" event={"ID":"ad917029-10ba-40c9-9b87-988c39079a64","Type":"ContainerDied","Data":"f326d3007ca6b59d57d59320b084c36a92298ae65a10c7bbea0588b94e6ecdaf"}
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.402222    5074 scope.go:122] "RemoveContainer" containerID="f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e"
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.405139    5074 generic.go:356] "Generic (PLEG): container finished" podID="36656517-378e-4358-b97f-1fd76762cf94" containerID="ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0" exitCode=0
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.405327    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" event={"ID":"36656517-378e-4358-b97f-1fd76762cf94","Type":"ContainerDied","Data":"ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0"}
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.405374    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs" event={"ID":"36656517-378e-4358-b97f-1fd76762cf94","Type":"ContainerDied","Data":"8ecae5f9f9b18ed2f0958a5670a0de21063d10e6b3b9b75899f02d2ef5a9cdbe"}
Nov 04 00:35:04.410825 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.406937    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"
Nov 04 00:35:04.418039 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.417993    5074 scope.go:122] "RemoveContainer" containerID="f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e"
Nov 04 00:35:04.418429 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:04.418398    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e\": container with ID starting with f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e not found: ID does not exist" containerID="f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e"
Nov 04 00:35:04.418543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.418434    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e"} err="failed to get container status \"f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e\": rpc error: code = NotFound desc = could not find container \"f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e\": container with ID starting with f7e0269559a63d7b06afe305d421313f03d0f4e1470d01968d81148f0dd5d91e not found: ID does not exist"
Nov 04 00:35:04.418543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.418461    5074 scope.go:122] "RemoveContainer" containerID="ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0"
Nov 04 00:35:04.431779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.431731    5074 scope.go:122] "RemoveContainer" containerID="ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0"
Nov 04 00:35:04.432045 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:04.432025    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0\": container with ID starting with ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0 not found: ID does not exist" containerID="ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0"
Nov 04 00:35:04.432107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.432051    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0"} err="failed to get container status \"ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0\": rpc error: code = NotFound desc = could not find container \"ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0\": container with ID starting with ad025ed521e967656c5ab83eff996cc87bfe256953352beada25b3e9a94303c0 not found: ID does not exist"
Nov 04 00:35:04.449251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.449223    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"]
Nov 04 00:35:04.469759 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.469711    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-86deca-76cb5ff96b-2cvvt"]
Nov 04 00:35:04.486331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.486307    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"]
Nov 04 00:35:04.495128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:04.495105    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-86deca-558755596c-2tsfs"]
Nov 04 00:35:05.053626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:05.053591    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36656517-378e-4358-b97f-1fd76762cf94" path="/var/lib/kubelet/pods/36656517-378e-4358-b97f-1fd76762cf94/volumes"
Nov 04 00:35:05.054190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:05.054125    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad917029-10ba-40c9-9b87-988c39079a64" path="/var/lib/kubelet/pods/ad917029-10ba-40c9-9b87-988c39079a64/volumes"
Nov 04 00:35:07.988895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:07.988863    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"]
Nov 04 00:35:08.008567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:08.008534    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"]
Nov 04 00:35:09.228073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228030    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7314-1bf12e"]
Nov 04 00:35:09.228815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228795    5074 cpu_manager.go:414] "containerMap: removing container" podUID="2e436f23-780c-4727-8e27-9120f9d533f8" containerName="main"
Nov 04 00:35:09.228815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228814    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e436f23-780c-4727-8e27-9120f9d533f8" containerName="main"
Nov 04 00:35:09.228929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228828    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ce218f33-c0f0-428e-9f5b-e30bbe253b61" containerName="revoke-keys"
Nov 04 00:35:09.228929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228834    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce218f33-c0f0-428e-9f5b-e30bbe253b61" containerName="revoke-keys"
Nov 04 00:35:09.228929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228860    5074 cpu_manager.go:414] "containerMap: removing container" podUID="36656517-378e-4358-b97f-1fd76762cf94" containerName="payload-pre-processing"
Nov 04 00:35:09.228929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228865    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36656517-378e-4358-b97f-1fd76762cf94" containerName="payload-pre-processing"
Nov 04 00:35:09.228929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228875    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ad917029-10ba-40c9-9b87-988c39079a64" containerName="payload-processing"
Nov 04 00:35:09.228929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228880    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad917029-10ba-40c9-9b87-988c39079a64" containerName="payload-processing"
Nov 04 00:35:09.229141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228981    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e436f23-780c-4727-8e27-9120f9d533f8" containerName="main"
Nov 04 00:35:09.229141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.228990    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce218f33-c0f0-428e-9f5b-e30bbe253b61" containerName="revoke-keys"
Nov 04 00:35:09.229141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.229000    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="36656517-378e-4358-b97f-1fd76762cf94" containerName="payload-pre-processing"
Nov 04 00:35:09.229141 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.229007    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad917029-10ba-40c9-9b87-988c39079a64" containerName="payload-processing"
Nov 04 00:35:09.238763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.236285    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:09.238763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.236643    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7314-1bf12e"]
Nov 04 00:35:09.242429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.240993    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hxqlx\""
Nov 04 00:35:09.305289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.305251    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wgr\" (UniqueName: \"kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr\") pod \"test-curl-7314-1bf12e\" (UID: \"c72c6a18-49de-4cbe-9048-bf8a75c6ff95\") " pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:09.406552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.406517    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wgr\" (UniqueName: \"kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr\") pod \"test-curl-7314-1bf12e\" (UID: \"c72c6a18-49de-4cbe-9048-bf8a75c6ff95\") " pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:09.421544 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.421511    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wgr\" (UniqueName: \"kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr\") pod \"test-curl-7314-1bf12e\" (UID: \"c72c6a18-49de-4cbe-9048-bf8a75c6ff95\") " pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:09.556103 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.556022    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:09.845640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:09.845600    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7314-1bf12e"]
Nov 04 00:35:10.456188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:10.456142    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7314-1bf12e" event={"ID":"c72c6a18-49de-4cbe-9048-bf8a75c6ff95","Type":"ContainerStarted","Data":"e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a"}
Nov 04 00:35:10.456667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:10.456199    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7314-1bf12e" event={"ID":"c72c6a18-49de-4cbe-9048-bf8a75c6ff95","Type":"ContainerStarted","Data":"51eac6297ade911df15301b765a3d741c5172ebb4a529db1734a966073a82fed"}
Nov 04 00:35:10.476064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:10.476008    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7314-1bf12e" podStartSLOduration=1.475994887 podStartE2EDuration="1.475994887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:35:10.473073015 +0000 UTC m=+3975.572673159" watchObservedRunningTime="2026-11-04 00:35:10.475994887 +0000 UTC m=+3975.575595031"
Nov 04 00:35:11.996554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:11.996514    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"]
Nov 04 00:35:12.440984 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.440939    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr" podUID="f0a9039b-60d1-484d-b01e-716f1def4cf7" containerName="main" containerID="cri-o://e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b" gracePeriod=45
Nov 04 00:35:12.719245 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.719214    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:35:12.860459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860420    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir\") pod \"f0a9039b-60d1-484d-b01e-716f1def4cf7\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") "
Nov 04 00:35:12.860459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860460    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs\") pod \"f0a9039b-60d1-484d-b01e-716f1def4cf7\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") "
Nov 04 00:35:12.860730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860492    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home\") pod \"f0a9039b-60d1-484d-b01e-716f1def4cf7\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") "
Nov 04 00:35:12.860730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860528    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2\" (UniqueName: \"kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2\") pod \"f0a9039b-60d1-484d-b01e-716f1def4cf7\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") "
Nov 04 00:35:12.860730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860556    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm\") pod \"f0a9039b-60d1-484d-b01e-716f1def4cf7\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") "
Nov 04 00:35:12.860730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860631    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache\") pod \"f0a9039b-60d1-484d-b01e-716f1def4cf7\" (UID: \"f0a9039b-60d1-484d-b01e-716f1def4cf7\") "
Nov 04 00:35:12.861021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860828    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir" pod "f0a9039b-60d1-484d-b01e-716f1def4cf7" (UID: "f0a9039b-60d1-484d-b01e-716f1def4cf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:12.861021 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.860887    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home" pod "f0a9039b-60d1-484d-b01e-716f1def4cf7" (UID: "f0a9039b-60d1-484d-b01e-716f1def4cf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:12.861130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.861058    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache" pod "f0a9039b-60d1-484d-b01e-716f1def4cf7" (UID: "f0a9039b-60d1-484d-b01e-716f1def4cf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:12.861254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.861234    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:12.861318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.861258    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:12.861318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.861273    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:12.862870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.862832    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2" pod "f0a9039b-60d1-484d-b01e-716f1def4cf7" (UID: "f0a9039b-60d1-484d-b01e-716f1def4cf7"). InnerVolumeSpecName "kube-api-access-68zb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:12.863168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.863153    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs" pod "f0a9039b-60d1-484d-b01e-716f1def4cf7" (UID: "f0a9039b-60d1-484d-b01e-716f1def4cf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:12.863225 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.863210    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm" pod "f0a9039b-60d1-484d-b01e-716f1def4cf7" (UID: "f0a9039b-60d1-484d-b01e-716f1def4cf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:12.962264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.962181    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a9039b-60d1-484d-b01e-716f1def4cf7-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:12.962264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.962215    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zb2\" (UniqueName: \"kubernetes.io/projected/f0a9039b-60d1-484d-b01e-716f1def4cf7-kube-api-access-68zb2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:12.962264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.962229    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a9039b-60d1-484d-b01e-716f1def4cf7-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:12.989526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:12.989484    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642" podUID="735b69d5-f762-4c20-b6c0-45c16656e2ac" containerName="payload-processing" containerID="cri-o://e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c" gracePeriod=25
Nov 04 00:35:13.009223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.009187    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerName="payload-pre-processing" containerID="cri-o://b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a" gracePeriod=25
Nov 04 00:35:13.242727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.242702    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:35:13.366894 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.366856    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm\" (UniqueName: \"kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm\") pod \"735b69d5-f762-4c20-b6c0-45c16656e2ac\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") "
Nov 04 00:35:13.367087 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.366999    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume\") pod \"735b69d5-f762-4c20-b6c0-45c16656e2ac\" (UID: \"735b69d5-f762-4c20-b6c0-45c16656e2ac\") "
Nov 04 00:35:13.367568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.367491    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume" pod "735b69d5-f762-4c20-b6c0-45c16656e2ac" (UID: "735b69d5-f762-4c20-b6c0-45c16656e2ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:13.369258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.369230    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm" pod "735b69d5-f762-4c20-b6c0-45c16656e2ac" (UID: "735b69d5-f762-4c20-b6c0-45c16656e2ac"). InnerVolumeSpecName "kube-api-access-ndbrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:13.469180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.469150    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbrm\" (UniqueName: \"kubernetes.io/projected/735b69d5-f762-4c20-b6c0-45c16656e2ac-kube-api-access-ndbrm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:13.469180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.469181    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735b69d5-f762-4c20-b6c0-45c16656e2ac-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:13.480352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.480323    5074 generic.go:356] "Generic (PLEG): container finished" podID="735b69d5-f762-4c20-b6c0-45c16656e2ac" containerID="e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c" exitCode=0
Nov 04 00:35:13.480508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.480380    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"
Nov 04 00:35:13.480508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.480407    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642" event={"ID":"735b69d5-f762-4c20-b6c0-45c16656e2ac","Type":"ContainerDied","Data":"e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c"}
Nov 04 00:35:13.480508 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.480491    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642" event={"ID":"735b69d5-f762-4c20-b6c0-45c16656e2ac","Type":"ContainerDied","Data":"fa2387e91f8391c2e70d4dd0f1adb9b751fbb615fa8850ad0d56f5c9cbd73c01"}
Nov 04 00:35:13.480688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.480522    5074 scope.go:122] "RemoveContainer" containerID="e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c"
Nov 04 00:35:13.482512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.482202    5074 generic.go:356] "Generic (PLEG): container finished" podID="f0a9039b-60d1-484d-b01e-716f1def4cf7" containerID="e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b" exitCode=0
Nov 04 00:35:13.482512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.482257    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr" event={"ID":"f0a9039b-60d1-484d-b01e-716f1def4cf7","Type":"ContainerDied","Data":"e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b"}
Nov 04 00:35:13.482512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.482285    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"
Nov 04 00:35:13.482512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.482302    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr" event={"ID":"f0a9039b-60d1-484d-b01e-716f1def4cf7","Type":"ContainerDied","Data":"b29907971b96ebba98d142adcdff73608ceee7ced227ca61ff88b5a6a765f167"}
Nov 04 00:35:13.493883 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.493867    5074 scope.go:122] "RemoveContainer" containerID="e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c"
Nov 04 00:35:13.494174 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:13.494149    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c\": container with ID starting with e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c not found: ID does not exist" containerID="e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c"
Nov 04 00:35:13.494233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.494184    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c"} err="failed to get container status \"e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c\": rpc error: code = NotFound desc = could not find container \"e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c\": container with ID starting with e3bd4f96f8684c686247ea640143c712cd99bbc17aa61416b20d3424d880ed6c not found: ID does not exist"
Nov 04 00:35:13.494233 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.494209    5074 scope.go:122] "RemoveContainer" containerID="e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b"
Nov 04 00:35:13.506196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.506164    5074 scope.go:122] "RemoveContainer" containerID="e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b"
Nov 04 00:35:13.506357 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.506340    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"]
Nov 04 00:35:13.506500 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:13.506478    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b\": container with ID starting with e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b not found: ID does not exist" containerID="e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b"
Nov 04 00:35:13.506555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.506510    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b"} err="failed to get container status \"e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b\": rpc error: code = NotFound desc = could not find container \"e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b\": container with ID starting with e8728845e810b12c99b6b131642ae56bfcc817e4a614f6a292d7215669932d4b not found: ID does not exist"
Nov 04 00:35:13.509861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.509841    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4331a7/rate-test-model-4331a7-kserve-6f79949649-wpsxr"]
Nov 04 00:35:13.517796 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.517777    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"]
Nov 04 00:35:13.522126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:13.522104    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fa0d0a-797759947b-5g642"]
Nov 04 00:35:15.019593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:15.019559    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735b69d5-f762-4c20-b6c0-45c16656e2ac" path="/var/lib/kubelet/pods/735b69d5-f762-4c20-b6c0-45c16656e2ac/volumes"
Nov 04 00:35:15.020098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:15.020081    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a9039b-60d1-484d-b01e-716f1def4cf7" path="/var/lib/kubelet/pods/f0a9039b-60d1-484d-b01e-716f1def4cf7/volumes"
Nov 04 00:35:16.997493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:16.997453    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" podUID="1ac22190-7677-4194-9c52-47b04738f995" containerName="payload-processing" containerID="cri-o://e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce" gracePeriod=25
Nov 04 00:35:17.141803 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.141755    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" podUID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" containerName="main" containerID="cri-o://842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9" gracePeriod=45
Nov 04 00:35:17.171078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.171039    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" podUID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" containerName="main" probeResult="failure" output="Get \"https://10.132.1.165:8000/ready\": dial tcp 10.132.1.165:8000: connect: connection refused"
Nov 04 00:35:17.325450 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.325416    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:35:17.423356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.423317    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume\") pod \"1ac22190-7677-4194-9c52-47b04738f995\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") "
Nov 04 00:35:17.423571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.423407    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx\" (UniqueName: \"kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx\") pod \"1ac22190-7677-4194-9c52-47b04738f995\" (UID: \"1ac22190-7677-4194-9c52-47b04738f995\") "
Nov 04 00:35:17.424579 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.424542    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume" pod "1ac22190-7677-4194-9c52-47b04738f995" (UID: "1ac22190-7677-4194-9c52-47b04738f995"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:17.427877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.427841    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx" pod "1ac22190-7677-4194-9c52-47b04738f995" (UID: "1ac22190-7677-4194-9c52-47b04738f995"). InnerVolumeSpecName "kube-api-access-hf6xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:17.451542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.451517    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:35:17.528187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528100    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm\") pod \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") "
Nov 04 00:35:17.528187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528172    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs\") pod \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") "
Nov 04 00:35:17.528420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528216    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir\") pod \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") "
Nov 04 00:35:17.528420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528287    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52\" (UniqueName: \"kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52\") pod \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") "
Nov 04 00:35:17.528571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528538    5074 generic.go:356] "Generic (PLEG): container finished" podID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" containerID="842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9" exitCode=0
Nov 04 00:35:17.528966 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528934    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache" pod "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" (UID: "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:17.529183 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.529105    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir" pod "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" (UID: "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:17.529319 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528599    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache\") pod \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") "
Nov 04 00:35:17.529533 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.529008    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"
Nov 04 00:35:17.529826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.528977    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" event={"ID":"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a","Type":"ContainerDied","Data":"842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9"}
Nov 04 00:35:17.529826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.529691    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84" event={"ID":"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a","Type":"ContainerDied","Data":"db76d7f90bf3ef9d6b05361a9970fe604d3572925fcda61519d8e572a422ea6e"}
Nov 04 00:35:17.529826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.529765    5074 scope.go:122] "RemoveContainer" containerID="842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9"
Nov 04 00:35:17.530571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.530221    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home" pod "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" (UID: "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:17.531365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.531023    5074 generic.go:356] "Generic (PLEG): container finished" podID="1ac22190-7677-4194-9c52-47b04738f995" containerID="e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce" exitCode=0
Nov 04 00:35:17.531365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.531035    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm" pod "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" (UID: "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:17.531365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.531127    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"
Nov 04 00:35:17.531365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.531146    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" event={"ID":"1ac22190-7677-4194-9c52-47b04738f995","Type":"ContainerDied","Data":"e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce"}
Nov 04 00:35:17.531365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.531211    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" event={"ID":"1ac22190-7677-4194-9c52-47b04738f995","Type":"ContainerDied","Data":"5b2241f755799ea192a7a731680a3c98125fa30dc12f4a2c311fa2ccbd3900ac"}
Nov 04 00:35:17.531365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.531211    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52" pod "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" (UID: "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a"). InnerVolumeSpecName "kube-api-access-mdh52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:17.531785 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.529525    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home\") pod \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\" (UID: \"22c2d6be-9efd-4f5d-80a1-b5b196a57b8a\") "
Nov 04 00:35:17.532348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532196    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf6xx\" (UniqueName: \"kubernetes.io/projected/1ac22190-7677-4194-9c52-47b04738f995-kube-api-access-hf6xx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.532348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532222    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.532348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532235    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.532348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532248    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.532348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532260    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.532348 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532273    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac22190-7677-4194-9c52-47b04738f995-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.532840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.532813    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdh52\" (UniqueName: \"kubernetes.io/projected/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-kube-api-access-mdh52\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.535600 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.534169    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs" pod "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" (UID: "22c2d6be-9efd-4f5d-80a1-b5b196a57b8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:17.543680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.543657    5074 scope.go:122] "RemoveContainer" containerID="842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9"
Nov 04 00:35:17.544117 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:17.544094    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9\": container with ID starting with 842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9 not found: ID does not exist" containerID="842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9"
Nov 04 00:35:17.544229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.544127    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9"} err="failed to get container status \"842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9\": rpc error: code = NotFound desc = could not find container \"842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9\": container with ID starting with 842d02853400c9ae66f79bfa7a881d1de1b2b5d4a34de0dcba793542aaad91c9 not found: ID does not exist"
Nov 04 00:35:17.544229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.544153    5074 scope.go:122] "RemoveContainer" containerID="e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce"
Nov 04 00:35:17.553842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.553822    5074 scope.go:122] "RemoveContainer" containerID="e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce"
Nov 04 00:35:17.554088 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:17.554070    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce\": container with ID starting with e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce not found: ID does not exist" containerID="e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce"
Nov 04 00:35:17.554145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.554093    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce"} err="failed to get container status \"e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce\": rpc error: code = NotFound desc = could not find container \"e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce\": container with ID starting with e8e8103752b6dde7d8838cb957b00febe4a9ba11265da9f4b1cb740279de46ce not found: ID does not exist"
Nov 04 00:35:17.560479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.560455    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"]
Nov 04 00:35:17.564402 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.564381    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l"]
Nov 04 00:35:17.633750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.633710    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:17.876150 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.876116    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:17.891349 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.891319    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"]
Nov 04 00:35:17.899347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:17.899321    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-398158/rate-test-model-398158-kserve-6d58775ff7-52s84"]
Nov 04 00:35:18.197136 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:18.197090    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4376d-966d67f66-bmj7l" podUID="1ac22190-7677-4194-9c52-47b04738f995" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.243:50052\" within 1s: context deadline exceeded"
Nov 04 00:35:18.197557 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:35:18.197203    5074 logging.go:55] [core] [Channel #25540 SubChannel #25541]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.243:50052", ServerName: "10.132.0.243:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.243:50052: operation was canceled"
Nov 04 00:35:19.067149 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:19.067111    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac22190-7677-4194-9c52-47b04738f995" path="/var/lib/kubelet/pods/1ac22190-7677-4194-9c52-47b04738f995/volumes"
Nov 04 00:35:19.067735 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:19.067713    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" path="/var/lib/kubelet/pods/22c2d6be-9efd-4f5d-80a1-b5b196a57b8a/volumes"
Nov 04 00:35:21.226477 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.226441    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7314-1bf12e"]
Nov 04 00:35:21.227041 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.226636    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7314-1bf12e" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" containerName="test-curl-7314-1bf12e" containerID="cri-o://e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a" gracePeriod=2
Nov 04 00:35:21.231553 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:21.231518    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:21.233019 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.232956    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7314-1bf12e"]
Nov 04 00:35:21.380460 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.380428    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7314-98e93f"]
Nov 04 00:35:21.381017 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381002    5074 cpu_manager.go:414] "containerMap: removing container" podUID="735b69d5-f762-4c20-b6c0-45c16656e2ac" containerName="payload-processing"
Nov 04 00:35:21.381070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381019    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735b69d5-f762-4c20-b6c0-45c16656e2ac" containerName="payload-processing"
Nov 04 00:35:21.381070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381038    5074 cpu_manager.go:414] "containerMap: removing container" podUID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" containerName="main"
Nov 04 00:35:21.381070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381043    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" containerName="main"
Nov 04 00:35:21.381070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381061    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1ac22190-7677-4194-9c52-47b04738f995" containerName="payload-processing"
Nov 04 00:35:21.381070 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381067    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac22190-7677-4194-9c52-47b04738f995" containerName="payload-processing"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381079    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" containerName="test-curl-7314-1bf12e"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381084    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" containerName="test-curl-7314-1bf12e"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381094    5074 cpu_manager.go:414] "containerMap: removing container" podUID="f0a9039b-60d1-484d-b01e-716f1def4cf7" containerName="main"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381099    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a9039b-60d1-484d-b01e-716f1def4cf7" containerName="main"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381177    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" containerName="test-curl-7314-1bf12e"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381185    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a9039b-60d1-484d-b01e-716f1def4cf7" containerName="main"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381196    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="735b69d5-f762-4c20-b6c0-45c16656e2ac" containerName="payload-processing"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381204    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c2d6be-9efd-4f5d-80a1-b5b196a57b8a" containerName="main"
Nov 04 00:35:21.381244 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.381212    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac22190-7677-4194-9c52-47b04738f995" containerName="payload-processing"
Nov 04 00:35:21.386301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.386276    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:21.388352 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.388326    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7314-98e93f"]
Nov 04 00:35:21.412344 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:21.412314    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:21.484620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.484540    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8lx\" (UniqueName: \"kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx\") pod \"test-curl-7314-98e93f\" (UID: \"df4646ce-554c-4980-bda1-2554938cfcfd\") " pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:21.586223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.586186    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8lx\" (UniqueName: \"kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx\") pod \"test-curl-7314-98e93f\" (UID: \"df4646ce-554c-4980-bda1-2554938cfcfd\") " pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:21.599631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.599594    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8lx\" (UniqueName: \"kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx\") pod \"test-curl-7314-98e93f\" (UID: \"df4646ce-554c-4980-bda1-2554938cfcfd\") " pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:21.710754 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.710699    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:21.992685 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:21.992652    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7314-98e93f"]
Nov 04 00:35:21.993803 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:35:21.993777    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf4646ce_554c_4980_bda1_2554938cfcfd.slice/crio-75c1a75e6d253b5e79ddfd704117f407fe13ed57c47ae8b93f12eb8c699a0f9e WatchSource:0}: Error finding container 75c1a75e6d253b5e79ddfd704117f407fe13ed57c47ae8b93f12eb8c699a0f9e: Status 404 returned error can't find the container with id 75c1a75e6d253b5e79ddfd704117f407fe13ed57c47ae8b93f12eb8c699a0f9e
Nov 04 00:35:22.573375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:22.573333    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7314-98e93f" event={"ID":"df4646ce-554c-4980-bda1-2554938cfcfd","Type":"ContainerStarted","Data":"52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d"}
Nov 04 00:35:22.573779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:22.573384    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7314-98e93f" event={"ID":"df4646ce-554c-4980-bda1-2554938cfcfd","Type":"ContainerStarted","Data":"75c1a75e6d253b5e79ddfd704117f407fe13ed57c47ae8b93f12eb8c699a0f9e"}
Nov 04 00:35:22.592510 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:22.592453    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7314-98e93f" podStartSLOduration=1.5924391089999999 podStartE2EDuration="1.592439109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:35:22.59101863 +0000 UTC m=+3987.690618768" watchObservedRunningTime="2026-11-04 00:35:22.592439109 +0000 UTC m=+3987.692039254"
Nov 04 00:35:22.594168 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:22.594140    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:23.314083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.314051    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7314-98e93f"]
Nov 04 00:35:23.323218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.323188    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7314-98e93f"]
Nov 04 00:35:23.447377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.447346    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:23.508884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.508806    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr\" (UniqueName: \"kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr\") pod \"c72c6a18-49de-4cbe-9048-bf8a75c6ff95\" (UID: \"c72c6a18-49de-4cbe-9048-bf8a75c6ff95\") "
Nov 04 00:35:23.510972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.510945    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr" pod "c72c6a18-49de-4cbe-9048-bf8a75c6ff95" (UID: "c72c6a18-49de-4cbe-9048-bf8a75c6ff95"). InnerVolumeSpecName "kube-api-access-t4wgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:23.581838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.581799    5074 generic.go:356] "Generic (PLEG): container finished" podID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" containerID="e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a" exitCode=137
Nov 04 00:35:23.582285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.581850    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:23.582285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.581895    5074 scope.go:122] "RemoveContainer" containerID="e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a"
Nov 04 00:35:23.592051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.592027    5074 scope.go:122] "RemoveContainer" containerID="e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a"
Nov 04 00:35:23.592360 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:23.592337    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a\": container with ID starting with e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a not found: ID does not exist" containerID="e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a"
Nov 04 00:35:23.592408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.592368    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a"} err="failed to get container status \"e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a\": rpc error: code = NotFound desc = could not find container \"e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a\": container with ID starting with e61af7546ffab05096b7b9cfba3d8f9c3b862801a04194fbdbfbcd9deea8853a not found: ID does not exist"
Nov 04 00:35:23.594149 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:23.594112    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:23.609801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.609776    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wgr\" (UniqueName: \"kubernetes.io/projected/c72c6a18-49de-4cbe-9048-bf8a75c6ff95-kube-api-access-t4wgr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:23.836366 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.836285    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"]
Nov 04 00:35:23.837180 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.837158    5074 cpu_manager.go:414] "containerMap: removing container" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" containerName="test-curl-7314-98e93f"
Nov 04 00:35:23.837231 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.837184    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" containerName="test-curl-7314-98e93f"
Nov 04 00:35:23.837375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.837363    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" containerName="test-curl-7314-98e93f"
Nov 04 00:35:23.842885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.842855    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:23.847677 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:23.846963    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:23.847845 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.847825    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"]
Nov 04 00:35:23.848934 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.848911    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:35:23.849131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.849107    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:35:23.913708 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.913672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:23.913878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.913843    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blnf\" (UniqueName: \"kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:23.913961 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:23.913942    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.014818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.014779    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.015028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.014856    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blnf\" (UniqueName: \"kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.015028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.014913    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.015571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.015541    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.017608 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.017580    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.032153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.032130    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blnf\" (UniqueName: \"kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf\") pod \"maas-api-revoke-sub-59c26a1a-zz8mk\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.170013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.169916    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:24.462126 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:35:24.462094    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045116e7_557f_406c_987b_e2659b6014ec.slice/crio-57ac7035597bf29679d5dda134f832e404e66d73a49cfe341cbc993434e01c4c WatchSource:0}: Error finding container 57ac7035597bf29679d5dda134f832e404e66d73a49cfe341cbc993434e01c4c: Status 404 returned error can't find the container with id 57ac7035597bf29679d5dda134f832e404e66d73a49cfe341cbc993434e01c4c
Nov 04 00:35:24.462676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.462649    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"]
Nov 04 00:35:24.592010 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.591973    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk" event={"ID":"045116e7-557f-406c-987b-e2659b6014ec","Type":"ContainerStarted","Data":"57ac7035597bf29679d5dda134f832e404e66d73a49cfe341cbc993434e01c4c"}
Nov 04 00:35:24.592419 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:24.592061    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7314-98e93f" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" containerName="test-curl-7314-98e93f" containerID="cri-o://52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d" gracePeriod=2
Nov 04 00:35:24.595612 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:24.595587    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-98e93f\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:24.598940 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:24.598906    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:25.018907 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:25.018804    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-98e93f\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:25.020270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:25.020241    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" path="/var/lib/kubelet/pods/c72c6a18-49de-4cbe-9048-bf8a75c6ff95/volumes"
Nov 04 00:35:25.021981 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:25.021945    5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7314-1bf12e\" is forbidden: User \"system:node:ip-10-0-138-239.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-239.us-east-2.compute.internal' and this object" podUID="c72c6a18-49de-4cbe-9048-bf8a75c6ff95" pod="opendatahub/test-curl-7314-1bf12e"
Nov 04 00:35:25.600378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:25.600343    5074 generic.go:356] "Generic (PLEG): container finished" podID="045116e7-557f-406c-987b-e2659b6014ec" containerID="ee223b38a75589335628b089a7ed20a17266fb798d6f0a79dfc361686a4d3222" exitCode=0
Nov 04 00:35:25.600869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:25.600428    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk" event={"ID":"045116e7-557f-406c-987b-e2659b6014ec","Type":"ContainerDied","Data":"ee223b38a75589335628b089a7ed20a17266fb798d6f0a79dfc361686a4d3222"}
Nov 04 00:35:26.227045 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.227004    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"]
Nov 04 00:35:26.233563 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.233529    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"]
Nov 04 00:35:26.787384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.787355    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:26.835936 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.835903    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:26.854958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.854897    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca\") pod \"045116e7-557f-406c-987b-e2659b6014ec\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") "
Nov 04 00:35:26.855089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.855041    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf\" (UniqueName: \"kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf\") pod \"045116e7-557f-406c-987b-e2659b6014ec\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") "
Nov 04 00:35:26.855151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.855123    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca\") pod \"045116e7-557f-406c-987b-e2659b6014ec\" (UID: \"045116e7-557f-406c-987b-e2659b6014ec\") "
Nov 04 00:35:26.855426 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.855400    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca" pod "045116e7-557f-406c-987b-e2659b6014ec" (UID: "045116e7-557f-406c-987b-e2659b6014ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:26.855617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.855601    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045116e7-557f-406c-987b-e2659b6014ec-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:26.857294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.857271    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca" pod "045116e7-557f-406c-987b-e2659b6014ec" (UID: "045116e7-557f-406c-987b-e2659b6014ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:26.857729 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.857700    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf" pod "045116e7-557f-406c-987b-e2659b6014ec" (UID: "045116e7-557f-406c-987b-e2659b6014ec"). InnerVolumeSpecName "kube-api-access-2blnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:26.956812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.956778    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx\" (UniqueName: \"kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx\") pod \"df4646ce-554c-4980-bda1-2554938cfcfd\" (UID: \"df4646ce-554c-4980-bda1-2554938cfcfd\") "
Nov 04 00:35:26.957248 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.957224    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blnf\" (UniqueName: \"kubernetes.io/projected/045116e7-557f-406c-987b-e2659b6014ec-kube-api-access-2blnf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:26.957337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.957255    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045116e7-557f-406c-987b-e2659b6014ec-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:26.958884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:26.958860    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx" pod "df4646ce-554c-4980-bda1-2554938cfcfd" (UID: "df4646ce-554c-4980-bda1-2554938cfcfd"). InnerVolumeSpecName "kube-api-access-dv8lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:27.018229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.018195    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4646ce-554c-4980-bda1-2554938cfcfd" path="/var/lib/kubelet/pods/df4646ce-554c-4980-bda1-2554938cfcfd/volumes"
Nov 04 00:35:27.059134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.059099    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8lx\" (UniqueName: \"kubernetes.io/projected/df4646ce-554c-4980-bda1-2554938cfcfd-kube-api-access-dv8lx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:27.619128 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.619101    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"
Nov 04 00:35:27.619461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.619135    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk" event={"ID":"045116e7-557f-406c-987b-e2659b6014ec","Type":"ContainerDied","Data":"57ac7035597bf29679d5dda134f832e404e66d73a49cfe341cbc993434e01c4c"}
Nov 04 00:35:27.619461 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.619295    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ac7035597bf29679d5dda134f832e404e66d73a49cfe341cbc993434e01c4c"
Nov 04 00:35:27.620903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.620871    5074 generic.go:356] "Generic (PLEG): container finished" podID="df4646ce-554c-4980-bda1-2554938cfcfd" containerID="52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d" exitCode=137
Nov 04 00:35:27.621044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.620925    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7314-98e93f"
Nov 04 00:35:27.621044 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.620958    5074 scope.go:122] "RemoveContainer" containerID="52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d"
Nov 04 00:35:27.641851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.641819    5074 scope.go:122] "RemoveContainer" containerID="52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d"
Nov 04 00:35:27.642255 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:27.642226    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d\": container with ID starting with 52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d not found: ID does not exist" containerID="52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d"
Nov 04 00:35:27.642382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.642263    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d"} err="failed to get container status \"52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d\": rpc error: code = NotFound desc = could not find container \"52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d\": container with ID starting with 52047bb7ded1b8d8f8ae5e87bc946a353232a89736ebcb3c7082b6d4e709a22d not found: ID does not exist"
Nov 04 00:35:27.876542 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:27.876455    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:28.023470 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:28.023428    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:35:31.228210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.228161    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj" podUID="66eb257f-677e-4efa-be06-5466c93cc0cc" containerName="payload-processing" containerID="cri-o://e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25" gracePeriod=25
Nov 04 00:35:31.234800 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.234729    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" podUID="8f550104-0924-4be3-b048-da09e218369b" containerName="payload-pre-processing" containerID="cri-o://c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09" gracePeriod=25
Nov 04 00:35:31.506539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.506512    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:35:31.616210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.616177    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume\") pod \"66eb257f-677e-4efa-be06-5466c93cc0cc\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") "
Nov 04 00:35:31.616400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.616291    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq\" (UniqueName: \"kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq\") pod \"66eb257f-677e-4efa-be06-5466c93cc0cc\" (UID: \"66eb257f-677e-4efa-be06-5466c93cc0cc\") "
Nov 04 00:35:31.616669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.616639    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume" pod "66eb257f-677e-4efa-be06-5466c93cc0cc" (UID: "66eb257f-677e-4efa-be06-5466c93cc0cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:31.617407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.616896    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eb257f-677e-4efa-be06-5466c93cc0cc-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:31.618727 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.618701    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq" pod "66eb257f-677e-4efa-be06-5466c93cc0cc" (UID: "66eb257f-677e-4efa-be06-5466c93cc0cc"). InnerVolumeSpecName "kube-api-access-hd8qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:31.658137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.658096    5074 generic.go:356] "Generic (PLEG): container finished" podID="66eb257f-677e-4efa-be06-5466c93cc0cc" containerID="e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25" exitCode=0
Nov 04 00:35:31.658137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.658137    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"
Nov 04 00:35:31.658369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.658175    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj" event={"ID":"66eb257f-677e-4efa-be06-5466c93cc0cc","Type":"ContainerDied","Data":"e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25"}
Nov 04 00:35:31.658369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.658237    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj" event={"ID":"66eb257f-677e-4efa-be06-5466c93cc0cc","Type":"ContainerDied","Data":"5c5afbcb2abef95c864d47ec7fd940348b8788b7da258fce6e160a73f9d667c0"}
Nov 04 00:35:31.658369 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.658261    5074 scope.go:122] "RemoveContainer" containerID="e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25"
Nov 04 00:35:31.669485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.669463    5074 scope.go:122] "RemoveContainer" containerID="e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25"
Nov 04 00:35:31.669774 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:31.669730    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25\": container with ID starting with e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25 not found: ID does not exist" containerID="e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25"
Nov 04 00:35:31.669912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.669774    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25"} err="failed to get container status \"e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25\": rpc error: code = NotFound desc = could not find container \"e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25\": container with ID starting with e7c750518b8f8b5c4ec6cc1cf42499d0d3cfd582a7cbb664b1b63b531e686e25 not found: ID does not exist"
Nov 04 00:35:31.692851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.692818    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"]
Nov 04 00:35:31.695965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.695939    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2697a0-577f5f8b48-kzczj"]
Nov 04 00:35:31.717712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:31.717671    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8qq\" (UniqueName: \"kubernetes.io/projected/66eb257f-677e-4efa-be06-5466c93cc0cc-kube-api-access-hd8qq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:33.019611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.019578    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66eb257f-677e-4efa-be06-5466c93cc0cc" path="/var/lib/kubelet/pods/66eb257f-677e-4efa-be06-5466c93cc0cc/volumes"
Nov 04 00:35:33.282567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.282504    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:35:33.334729 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.334688    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz\" (UniqueName: \"kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz\") pod \"4b1131b3-d78d-4434-9d46-8c2044aa02db\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") "
Nov 04 00:35:33.334960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.334924    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume\") pod \"4b1131b3-d78d-4434-9d46-8c2044aa02db\" (UID: \"4b1131b3-d78d-4434-9d46-8c2044aa02db\") "
Nov 04 00:35:33.335410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.335372    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume" pod "4b1131b3-d78d-4434-9d46-8c2044aa02db" (UID: "4b1131b3-d78d-4434-9d46-8c2044aa02db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:33.337565 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.337520    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz" pod "4b1131b3-d78d-4434-9d46-8c2044aa02db" (UID: "4b1131b3-d78d-4434-9d46-8c2044aa02db"). InnerVolumeSpecName "kube-api-access-j7dvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:33.436452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.436412    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1131b3-d78d-4434-9d46-8c2044aa02db-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:33.436452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.436449    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dvz\" (UniqueName: \"kubernetes.io/projected/4b1131b3-d78d-4434-9d46-8c2044aa02db-kube-api-access-j7dvz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:33.675537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.675443    5074 generic.go:356] "Generic (PLEG): container finished" podID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerID="b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a" exitCode=0
Nov 04 00:35:33.675537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.675491    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"
Nov 04 00:35:33.675537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.675527    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" event={"ID":"4b1131b3-d78d-4434-9d46-8c2044aa02db","Type":"ContainerDied","Data":"b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a"}
Nov 04 00:35:33.675888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.675583    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht" event={"ID":"4b1131b3-d78d-4434-9d46-8c2044aa02db","Type":"ContainerDied","Data":"e56086782bc96db365d1a4a7b00a8a81f1dd00dcf8774ce7b6429b6ed825cf06"}
Nov 04 00:35:33.675888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.675604    5074 scope.go:122] "RemoveContainer" containerID="b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a"
Nov 04 00:35:33.687129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.687102    5074 scope.go:122] "RemoveContainer" containerID="b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a"
Nov 04 00:35:33.687444 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:33.687418    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a\": container with ID starting with b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a not found: ID does not exist" containerID="b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a"
Nov 04 00:35:33.687511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.687457    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a"} err="failed to get container status \"b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a\": rpc error: code = NotFound desc = could not find container \"b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a\": container with ID starting with b5801e624c9a98a85543cc4c37332fd7454cfd3bd2f4ef55f11cbab53d96702a not found: ID does not exist"
Nov 04 00:35:33.704546 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.704518    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"]
Nov 04 00:35:33.708164 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:33.708141    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fa0d0a-557cc45c9f-l8lht"]
Nov 04 00:35:34.055509 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.055476    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"]
Nov 04 00:35:34.056122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056107    5074 cpu_manager.go:414] "containerMap: removing container" podUID="045116e7-557f-406c-987b-e2659b6014ec" containerName="revoke-keys"
Nov 04 00:35:34.056163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056123    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045116e7-557f-406c-987b-e2659b6014ec" containerName="revoke-keys"
Nov 04 00:35:34.056163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056136    5074 cpu_manager.go:414] "containerMap: removing container" podUID="66eb257f-677e-4efa-be06-5466c93cc0cc" containerName="payload-processing"
Nov 04 00:35:34.056163 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056143    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66eb257f-677e-4efa-be06-5466c93cc0cc" containerName="payload-processing"
Nov 04 00:35:34.056300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056196    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerName="payload-pre-processing"
Nov 04 00:35:34.056300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056202    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerName="payload-pre-processing"
Nov 04 00:35:34.056373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056313    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" containerName="payload-pre-processing"
Nov 04 00:35:34.056373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056325    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="045116e7-557f-406c-987b-e2659b6014ec" containerName="revoke-keys"
Nov 04 00:35:34.056373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.056332    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="66eb257f-677e-4efa-be06-5466c93cc0cc" containerName="payload-processing"
Nov 04 00:35:34.061308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.061288    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.065814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.065794    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:35:34.067574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.067551    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:35:34.068931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.068904    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"]
Nov 04 00:35:34.144481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.144440    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.144692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.144525    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.144692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.144576    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4964\" (UniqueName: \"kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.246229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.246190    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.246422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.246270    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.246422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.246312    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4964\" (UniqueName: \"kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.247111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.247087    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.248972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.248948    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.259847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.259820    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4964\" (UniqueName: \"kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964\") pod \"maas-api-revoke-sub-42672ff8-sscmz\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.381338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.381023    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:34.679827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:34.679797    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"]
Nov 04 00:35:34.681259 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:35:34.681230    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e291f68_9ec7_4ffc_9a88_cee4d262fddf.slice/crio-ca1918087168b7d90984443ca0e91d4b4551955156217526bb7f2ec1149df48e WatchSource:0}: Error finding container ca1918087168b7d90984443ca0e91d4b4551955156217526bb7f2ec1149df48e: Status 404 returned error can't find the container with id ca1918087168b7d90984443ca0e91d4b4551955156217526bb7f2ec1149df48e
Nov 04 00:35:35.020006 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:35.019978    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1131b3-d78d-4434-9d46-8c2044aa02db" path="/var/lib/kubelet/pods/4b1131b3-d78d-4434-9d46-8c2044aa02db/volumes"
Nov 04 00:35:35.693931 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:35.693899    5074 generic.go:356] "Generic (PLEG): container finished" podID="8e291f68-9ec7-4ffc-9a88-cee4d262fddf" containerID="70a95e1d6d28c08c322bbcb65c2b6be8f1fa1c3b59a2d0f64694295b56c126a3" exitCode=0
Nov 04 00:35:35.694358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:35.693994    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz" event={"ID":"8e291f68-9ec7-4ffc-9a88-cee4d262fddf","Type":"ContainerDied","Data":"70a95e1d6d28c08c322bbcb65c2b6be8f1fa1c3b59a2d0f64694295b56c126a3"}
Nov 04 00:35:35.694358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:35.694070    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz" event={"ID":"8e291f68-9ec7-4ffc-9a88-cee4d262fddf","Type":"ContainerStarted","Data":"ca1918087168b7d90984443ca0e91d4b4551955156217526bb7f2ec1149df48e"}
Nov 04 00:35:36.865981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.865947    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:36.980695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.980613    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca\") pod \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") "
Nov 04 00:35:36.980695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.980670    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca\") pod \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") "
Nov 04 00:35:36.980928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.980734    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964\" (UniqueName: \"kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964\") pod \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\" (UID: \"8e291f68-9ec7-4ffc-9a88-cee4d262fddf\") "
Nov 04 00:35:36.981102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.981074    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca" pod "8e291f68-9ec7-4ffc-9a88-cee4d262fddf" (UID: "8e291f68-9ec7-4ffc-9a88-cee4d262fddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:36.981253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.981237    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:36.982885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.982862    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca" pod "8e291f68-9ec7-4ffc-9a88-cee4d262fddf" (UID: "8e291f68-9ec7-4ffc-9a88-cee4d262fddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:36.983010 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:36.982934    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964" pod "8e291f68-9ec7-4ffc-9a88-cee4d262fddf" (UID: "8e291f68-9ec7-4ffc-9a88-cee4d262fddf"). InnerVolumeSpecName "kube-api-access-r4964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:37.082238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:37.082205    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4964\" (UniqueName: \"kubernetes.io/projected/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-kube-api-access-r4964\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:37.082238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:37.082237    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e291f68-9ec7-4ffc-9a88-cee4d262fddf-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:37.712612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:37.712581    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"
Nov 04 00:35:37.712851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:37.712576    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz" event={"ID":"8e291f68-9ec7-4ffc-9a88-cee4d262fddf","Type":"ContainerDied","Data":"ca1918087168b7d90984443ca0e91d4b4551955156217526bb7f2ec1149df48e"}
Nov 04 00:35:37.712851 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:37.712715    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1918087168b7d90984443ca0e91d4b4551955156217526bb7f2ec1149df48e"
Nov 04 00:35:39.057234 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:39.057192    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" podUID="8f550104-0924-4be3-b048-da09e218369b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:39.723944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:39.723900    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"]
Nov 04 00:35:39.724172 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:39.724141    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy" containerID="cri-o://d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2" gracePeriod=30
Nov 04 00:35:44.869435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.869400    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"]
Nov 04 00:35:44.870028 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.870005    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8e291f68-9ec7-4ffc-9a88-cee4d262fddf" containerName="revoke-keys"
Nov 04 00:35:44.870081 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.870029    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e291f68-9ec7-4ffc-9a88-cee4d262fddf" containerName="revoke-keys"
Nov 04 00:35:44.870134 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.870125    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e291f68-9ec7-4ffc-9a88-cee4d262fddf" containerName="revoke-keys"
Nov 04 00:35:44.890002 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.889965    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"]
Nov 04 00:35:44.892768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.892714    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:44.897983 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.897953    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:35:44.899795 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.899775    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:35:44.985001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.984965    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4kq\" (UniqueName: \"kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:44.985170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.985017    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:44.985170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:44.985144    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.039107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.039074    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:35:45.086190 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.086151    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.086373 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.086320    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4kq\" (UniqueName: \"kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.086429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.086382    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.087259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.087225    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.088794 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.088725    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.101433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.101398    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4kq\" (UniqueName: \"kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq\") pod \"maas-api-revoke-sub-fb77929a-fvdp4\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.187336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187297    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.187609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187345    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.187609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187396    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.187609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187418    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.187609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187437    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.187609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187501    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.187609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187604    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.188027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187630    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.188027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187667    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy\") pod \"67aa3221-e5de-45b7-b53d-329c239d6b7f\" (UID: \"67aa3221-e5de-45b7-b53d-329c239d6b7f\") "
Nov 04 00:35:45.188027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187731    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:45.188027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187793    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:45.188027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187813    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:45.188027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.187867    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:45.188378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.188056    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:45.188378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.188239    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa3221-e5de-45b7-b53d-329c239d6b7f-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.188378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.188259    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.188378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.188268    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.188378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.188277    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.188378 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.188289    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.189881 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.189858    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:45.189974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.189917    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:45.190197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.190168    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:45.190197 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.190187    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr" pod "67aa3221-e5de-45b7-b53d-329c239d6b7f" (UID: "67aa3221-e5de-45b7-b53d-329c239d6b7f"). InnerVolumeSpecName "kube-api-access-7zbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:45.214122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.214093    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:45.289585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.289552    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.289585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.289582    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.289844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.289593    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbtr\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-kube-api-access-7zbtr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.289844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.289602    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa3221-e5de-45b7-b53d-329c239d6b7f-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:45.499200 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.499165    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"]
Nov 04 00:35:45.500271 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:35:45.500243    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274ddccb_879b_4a79_b92a_e3846efaabb7.slice/crio-e49f0940cd10849ed14a7e97ae11f8dccd56c0560c57e607e72a2bf6ec0aa43f WatchSource:0}: Error finding container e49f0940cd10849ed14a7e97ae11f8dccd56c0560c57e607e72a2bf6ec0aa43f: Status 404 returned error can't find the container with id e49f0940cd10849ed14a7e97ae11f8dccd56c0560c57e607e72a2bf6ec0aa43f
Nov 04 00:35:45.784114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.784021    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4" event={"ID":"274ddccb-879b-4a79-b92a-e3846efaabb7","Type":"ContainerStarted","Data":"e49f0940cd10849ed14a7e97ae11f8dccd56c0560c57e607e72a2bf6ec0aa43f"}
Nov 04 00:35:45.785479 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.785451    5074 generic.go:356] "Generic (PLEG): container finished" podID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerID="d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2" exitCode=0
Nov 04 00:35:45.785619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.785507    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"
Nov 04 00:35:45.785619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.785538    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" event={"ID":"67aa3221-e5de-45b7-b53d-329c239d6b7f","Type":"ContainerDied","Data":"d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2"}
Nov 04 00:35:45.785619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.785597    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8" event={"ID":"67aa3221-e5de-45b7-b53d-329c239d6b7f","Type":"ContainerDied","Data":"0202ad6245cf5f81b58e402de5ef1e004851d0e52b8d07d24fd7194d63d8b230"}
Nov 04 00:35:45.785760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.785625    5074 scope.go:122] "RemoveContainer" containerID="d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2"
Nov 04 00:35:45.831878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.831854    5074 scope.go:122] "RemoveContainer" containerID="d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2"
Nov 04 00:35:45.832242 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:45.832221    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2\": container with ID starting with d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2 not found: ID does not exist" containerID="d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2"
Nov 04 00:35:45.832323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.832254    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2"} err="failed to get container status \"d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2\": rpc error: code = NotFound desc = could not find container \"d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2\": container with ID starting with d151614bd22abb356ae4086f2da6891f35fd91eff9758973451fe001927fc2b2 not found: ID does not exist"
Nov 04 00:35:45.844458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.844431    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"]
Nov 04 00:35:45.850616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:45.850589    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-l9nw8"]
Nov 04 00:35:46.796229 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:46.796194    5074 generic.go:356] "Generic (PLEG): container finished" podID="274ddccb-879b-4a79-b92a-e3846efaabb7" containerID="4166267c58757f9078cb847bda0a9a766554ea8304e24503251c2949e91a7e42" exitCode=0
Nov 04 00:35:46.796637 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:46.796229    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4" event={"ID":"274ddccb-879b-4a79-b92a-e3846efaabb7","Type":"ContainerDied","Data":"4166267c58757f9078cb847bda0a9a766554ea8304e24503251c2949e91a7e42"}
Nov 04 00:35:46.941687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:46.941646    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"]
Nov 04 00:35:46.949610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:46.949544    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"]
Nov 04 00:35:47.024499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:47.024460    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" path="/var/lib/kubelet/pods/67aa3221-e5de-45b7-b53d-329c239d6b7f/volumes"
Nov 04 00:35:47.976626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:47.976601    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:48.134673 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.134595    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca\") pod \"274ddccb-879b-4a79-b92a-e3846efaabb7\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") "
Nov 04 00:35:48.134865 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.134710    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq\" (UniqueName: \"kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq\") pod \"274ddccb-879b-4a79-b92a-e3846efaabb7\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") "
Nov 04 00:35:48.134865 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.134775    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca\") pod \"274ddccb-879b-4a79-b92a-e3846efaabb7\" (UID: \"274ddccb-879b-4a79-b92a-e3846efaabb7\") "
Nov 04 00:35:48.135418 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.135186    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca" pod "274ddccb-879b-4a79-b92a-e3846efaabb7" (UID: "274ddccb-879b-4a79-b92a-e3846efaabb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:48.137204 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.137179    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca" pod "274ddccb-879b-4a79-b92a-e3846efaabb7" (UID: "274ddccb-879b-4a79-b92a-e3846efaabb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:48.137555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.137532    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq" pod "274ddccb-879b-4a79-b92a-e3846efaabb7" (UID: "274ddccb-879b-4a79-b92a-e3846efaabb7"). InnerVolumeSpecName "kube-api-access-hv4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:48.236667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.236630    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:48.236667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.236665    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4kq\" (UniqueName: \"kubernetes.io/projected/274ddccb-879b-4a79-b92a-e3846efaabb7-kube-api-access-hv4kq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:48.236903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.236677    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274ddccb-879b-4a79-b92a-e3846efaabb7-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:48.814459 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.814428    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"
Nov 04 00:35:48.814653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.814459    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4" event={"ID":"274ddccb-879b-4a79-b92a-e3846efaabb7","Type":"ContainerDied","Data":"e49f0940cd10849ed14a7e97ae11f8dccd56c0560c57e607e72a2bf6ec0aa43f"}
Nov 04 00:35:48.814653 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:48.814519    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49f0940cd10849ed14a7e97ae11f8dccd56c0560c57e607e72a2bf6ec0aa43f"
Nov 04 00:35:49.057122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:49.057078    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" podUID="8f550104-0924-4be3-b048-da09e218369b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:51.535487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.535455    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:35:51.585631 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.585593    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95\" (UniqueName: \"kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95\") pod \"8f550104-0924-4be3-b048-da09e218369b\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") "
Nov 04 00:35:51.585867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.585666    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume\") pod \"8f550104-0924-4be3-b048-da09e218369b\" (UID: \"8f550104-0924-4be3-b048-da09e218369b\") "
Nov 04 00:35:51.586139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.586107    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume" pod "8f550104-0924-4be3-b048-da09e218369b" (UID: "8f550104-0924-4be3-b048-da09e218369b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:51.588519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.588491    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95" pod "8f550104-0924-4be3-b048-da09e218369b" (UID: "8f550104-0924-4be3-b048-da09e218369b"). InnerVolumeSpecName "kube-api-access-gpb95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:51.687394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.687352    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpb95\" (UniqueName: \"kubernetes.io/projected/8f550104-0924-4be3-b048-da09e218369b-kube-api-access-gpb95\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:51.687394 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.687387    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f550104-0924-4be3-b048-da09e218369b-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:51.835435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.835341    5074 generic.go:356] "Generic (PLEG): container finished" podID="8f550104-0924-4be3-b048-da09e218369b" containerID="c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09" exitCode=0
Nov 04 00:35:51.835435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.835397    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"
Nov 04 00:35:51.835435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.835408    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" event={"ID":"8f550104-0924-4be3-b048-da09e218369b","Type":"ContainerDied","Data":"c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09"}
Nov 04 00:35:51.835784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.835462    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s" event={"ID":"8f550104-0924-4be3-b048-da09e218369b","Type":"ContainerDied","Data":"97878703d2cd460d7c2715b8810ec3ea07d7905b7ab8742947a03c61efb40119"}
Nov 04 00:35:51.835784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.835485    5074 scope.go:122] "RemoveContainer" containerID="c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09"
Nov 04 00:35:51.848878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.848858    5074 scope.go:122] "RemoveContainer" containerID="c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09"
Nov 04 00:35:51.849180 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:51.849156    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09\": container with ID starting with c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09 not found: ID does not exist" containerID="c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09"
Nov 04 00:35:51.849249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.849190    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09"} err="failed to get container status \"c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09\": rpc error: code = NotFound desc = could not find container \"c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09\": container with ID starting with c37385d28773a4a1a478d8f1784c69f458ee2012ed0ec1df88578b9655ac6a09 not found: ID does not exist"
Nov 04 00:35:51.862616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.862596    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"]
Nov 04 00:35:51.865707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.865686    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2697a0-b4b99df9f-w2k4s"]
Nov 04 00:35:51.942596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.942547    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerName="payload-processing" containerID="cri-o://544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f" gracePeriod=25
Nov 04 00:35:51.950131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:51.950093    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerName="payload-pre-processing" containerID="cri-o://ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915" gracePeriod=25
Nov 04 00:35:53.017201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:53.017166    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f550104-0924-4be3-b048-da09e218369b" path="/var/lib/kubelet/pods/8f550104-0924-4be3-b048-da09e218369b/volumes"
Nov 04 00:35:55.306329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.306294    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"]
Nov 04 00:35:55.307107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307085    5074 cpu_manager.go:414] "containerMap: removing container" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy"
Nov 04 00:35:55.307232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307109    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy"
Nov 04 00:35:55.307232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307142    5074 cpu_manager.go:414] "containerMap: removing container" podUID="274ddccb-879b-4a79-b92a-e3846efaabb7" containerName="revoke-keys"
Nov 04 00:35:55.307232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307151    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274ddccb-879b-4a79-b92a-e3846efaabb7" containerName="revoke-keys"
Nov 04 00:35:55.307232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307177    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8f550104-0924-4be3-b048-da09e218369b" containerName="payload-pre-processing"
Nov 04 00:35:55.307232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307185    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f550104-0924-4be3-b048-da09e218369b" containerName="payload-pre-processing"
Nov 04 00:35:55.307493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307358    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa3221-e5de-45b7-b53d-329c239d6b7f" containerName="istio-proxy"
Nov 04 00:35:55.307493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307376    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="274ddccb-879b-4a79-b92a-e3846efaabb7" containerName="revoke-keys"
Nov 04 00:35:55.307493 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.307389    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f550104-0924-4be3-b048-da09e218369b" containerName="payload-pre-processing"
Nov 04 00:35:55.312462 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.312439    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.317622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.317596    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:35:55.319930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.319907    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:35:55.325232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.325202    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"]
Nov 04 00:35:55.332495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.332460    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.332837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.332812    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.332952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.332860    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8g4\" (UniqueName: \"kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.424617 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.424572    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:55.434488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.434454    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.434662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.434507    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8g4\" (UniqueName: \"kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.434662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.434653    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.435469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.435444    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.437334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.437311    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.447448 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.447423    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8g4\" (UniqueName: \"kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4\") pod \"maas-api-revoke-sub-fa46684a-85l95\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.630861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.630769    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:55.913170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:55.913137    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"]
Nov 04 00:35:55.914954 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:35:55.914926    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238dd50d_e199_4f30_8238_b22a43370116.slice/crio-f852a35999947cda267d431c25319664d9665bd3ec146bc1b5c263beabe51f8a WatchSource:0}: Error finding container f852a35999947cda267d431c25319664d9665bd3ec146bc1b5c263beabe51f8a: Status 404 returned error can't find the container with id f852a35999947cda267d431c25319664d9665bd3ec146bc1b5c263beabe51f8a
Nov 04 00:35:56.434438 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:56.434399    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:56.875228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:56.875193    5074 generic.go:356] "Generic (PLEG): container finished" podID="238dd50d-e199-4f30-8238-b22a43370116" containerID="5af308ed58e40eb9f52ca086308df614aafb6fe74b831284860c2af3416a1c4e" exitCode=0
Nov 04 00:35:56.875400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:56.875277    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95" event={"ID":"238dd50d-e199-4f30-8238-b22a43370116","Type":"ContainerDied","Data":"5af308ed58e40eb9f52ca086308df614aafb6fe74b831284860c2af3416a1c4e"}
Nov 04 00:35:56.875400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:56.875331    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95" event={"ID":"238dd50d-e199-4f30-8238-b22a43370116","Type":"ContainerStarted","Data":"f852a35999947cda267d431c25319664d9665bd3ec146bc1b5c263beabe51f8a"}
Nov 04 00:35:58.024594 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:35:58.024540    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:35:58.071208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.071180    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:58.167137 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.167058    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4\" (UniqueName: \"kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4\") pod \"238dd50d-e199-4f30-8238-b22a43370116\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") "
Nov 04 00:35:58.167296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.167165    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca\") pod \"238dd50d-e199-4f30-8238-b22a43370116\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") "
Nov 04 00:35:58.167296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.167207    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca\") pod \"238dd50d-e199-4f30-8238-b22a43370116\" (UID: \"238dd50d-e199-4f30-8238-b22a43370116\") "
Nov 04 00:35:58.167641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.167616    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca" pod "238dd50d-e199-4f30-8238-b22a43370116" (UID: "238dd50d-e199-4f30-8238-b22a43370116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:58.169207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.169183    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca" pod "238dd50d-e199-4f30-8238-b22a43370116" (UID: "238dd50d-e199-4f30-8238-b22a43370116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:58.169301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.169278    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4" pod "238dd50d-e199-4f30-8238-b22a43370116" (UID: "238dd50d-e199-4f30-8238-b22a43370116"). InnerVolumeSpecName "kube-api-access-ns8g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:58.268306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.268271    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238dd50d-e199-4f30-8238-b22a43370116-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:58.268306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.268303    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238dd50d-e199-4f30-8238-b22a43370116-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:58.268306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.268313    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8g4\" (UniqueName: \"kubernetes.io/projected/238dd50d-e199-4f30-8238-b22a43370116-kube-api-access-ns8g4\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:58.890504 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.890474    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"
Nov 04 00:35:58.890683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.890508    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95" event={"ID":"238dd50d-e199-4f30-8238-b22a43370116","Type":"ContainerDied","Data":"f852a35999947cda267d431c25319664d9665bd3ec146bc1b5c263beabe51f8a"}
Nov 04 00:35:58.890683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:35:58.890558    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f852a35999947cda267d431c25319664d9665bd3ec146bc1b5c263beabe51f8a"
Nov 04 00:36:01.599064 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:01.599026    5074 scope.go:122] "RemoveContainer" containerID="561aa0637aa6b10173d145abda447f8514b1287fa7a5059c30cc805e43409d30"
Nov 04 00:36:03.941090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:03.938939    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"]
Nov 04 00:36:03.941090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:03.939199    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" containerID="cri-o://caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060" gracePeriod=30
Nov 04 00:36:04.809034 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.808997    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"]
Nov 04 00:36:04.809573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.809559    5074 cpu_manager.go:414] "containerMap: removing container" podUID="238dd50d-e199-4f30-8238-b22a43370116" containerName="revoke-keys"
Nov 04 00:36:04.809641 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.809575    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="238dd50d-e199-4f30-8238-b22a43370116" containerName="revoke-keys"
Nov 04 00:36:04.809714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.809705    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="238dd50d-e199-4f30-8238-b22a43370116" containerName="revoke-keys"
Nov 04 00:36:04.814822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.814801    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.835026 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.834996    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:36:04.835396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.835373    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:36:04.836664 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.836637    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"]
Nov 04 00:36:04.844422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.844132    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktpz\" (UniqueName: \"kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.844422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.844176    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.844422 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.844234    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.944999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.944965    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktpz\" (UniqueName: \"kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.944999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.945003    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.945471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.945045    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.945849 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.945825    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.947668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.947640    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:04.958524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:04.958503    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktpz\" (UniqueName: \"kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz\") pod \"maas-api-revoke-sub-ca73e355-cgkkm\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:05.128476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.128382    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:05.413714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.413688    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"]
Nov 04 00:36:05.416219 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:36:05.416178    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9eaedd_71e2_41a1_b61f_61c385c290c7.slice/crio-1e5ba49b44d3382e2ae8fbc2b6b68f8e7bc9c1cf68b7d16671320898bd973134 WatchSource:0}: Error finding container 1e5ba49b44d3382e2ae8fbc2b6b68f8e7bc9c1cf68b7d16671320898bd973134: Status 404 returned error can't find the container with id 1e5ba49b44d3382e2ae8fbc2b6b68f8e7bc9c1cf68b7d16671320898bd973134
Nov 04 00:36:05.424903 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.424872    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:05.874810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.874768    5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:36:05.874999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.874832    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:36:05.912783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.912732    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"]
Nov 04 00:36:05.952523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.952482    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm" event={"ID":"9d9eaedd-71e2-41a1-b61f-61c385c290c7","Type":"ContainerStarted","Data":"df40ac677726a18e89c3a8c76abf459f00add9c9e53255d41b4ecc886181d070"}
Nov 04 00:36:05.953047 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:05.952557    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm" event={"ID":"9d9eaedd-71e2-41a1-b61f-61c385c290c7","Type":"ContainerStarted","Data":"1e5ba49b44d3382e2ae8fbc2b6b68f8e7bc9c1cf68b7d16671320898bd973134"}
Nov 04 00:36:06.026427 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:06.025628    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm" podStartSLOduration=2.025609941 podStartE2EDuration="2.025609941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:36:06.017686616 +0000 UTC m=+4031.117286759" watchObservedRunningTime="2026-11-04 00:36:06.025609941 +0000 UTC m=+4031.125210086"
Nov 04 00:36:06.434291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:06.434257    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:06.962683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:06.962646    5074 generic.go:356] "Generic (PLEG): container finished" podID="9d9eaedd-71e2-41a1-b61f-61c385c290c7" containerID="df40ac677726a18e89c3a8c76abf459f00add9c9e53255d41b4ecc886181d070" exitCode=0
Nov 04 00:36:06.963102 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:06.962735    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm" event={"ID":"9d9eaedd-71e2-41a1-b61f-61c385c290c7","Type":"ContainerDied","Data":"df40ac677726a18e89c3a8c76abf459f00add9c9e53255d41b4ecc886181d070"}
Nov 04 00:36:08.141083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.141052    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:08.182614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.182569    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca\") pod \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") "
Nov 04 00:36:08.183159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.182813    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz\" (UniqueName: \"kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz\") pod \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") "
Nov 04 00:36:08.183159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.182900    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca\") pod \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\" (UID: \"9d9eaedd-71e2-41a1-b61f-61c385c290c7\") "
Nov 04 00:36:08.183900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.183870    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca" pod "9d9eaedd-71e2-41a1-b61f-61c385c290c7" (UID: "9d9eaedd-71e2-41a1-b61f-61c385c290c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:08.189781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.185890    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz" pod "9d9eaedd-71e2-41a1-b61f-61c385c290c7" (UID: "9d9eaedd-71e2-41a1-b61f-61c385c290c7"). InnerVolumeSpecName "kube-api-access-5ktpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:08.189781 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.185993    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca" pod "9d9eaedd-71e2-41a1-b61f-61c385c290c7" (UID: "9d9eaedd-71e2-41a1-b61f-61c385c290c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:08.284770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.284672    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.284770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.284706    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktpz\" (UniqueName: \"kubernetes.io/projected/9d9eaedd-71e2-41a1-b61f-61c385c290c7-kube-api-access-5ktpz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.284770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.284721    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9eaedd-71e2-41a1-b61f-61c385c290c7-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.985854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.985810    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm" event={"ID":"9d9eaedd-71e2-41a1-b61f-61c385c290c7","Type":"ContainerDied","Data":"1e5ba49b44d3382e2ae8fbc2b6b68f8e7bc9c1cf68b7d16671320898bd973134"}
Nov 04 00:36:08.986005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.985890    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5ba49b44d3382e2ae8fbc2b6b68f8e7bc9c1cf68b7d16671320898bd973134"
Nov 04 00:36:08.986005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:08.985909    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"
Nov 04 00:36:09.156873 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.156835    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"]
Nov 04 00:36:09.299205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.299150    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:36:09.397024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.396991    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397075    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397128    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397219 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397196    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397244    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397310    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:09.397421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397334    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397379    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397425    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397473    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket\") pod \"89b34d86-f828-4b79-ac76-cac5c747ec69\" (UID: \"89b34d86-f828-4b79-ac76-cac5c747ec69\") "
Nov 04 00:36:09.397652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397500    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:09.397652 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397574    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:09.401784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.397977    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.401784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.398005    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b34d86-f828-4b79-ac76-cac5c747ec69-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.401784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.398028    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.401784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.398000    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:09.401784 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.398126    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:09.402249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.401903    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:09.402249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.401952    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:09.402249 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.401979    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "kube-api-access-7mmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:09.402978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.402951    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo" pod "89b34d86-f828-4b79-ac76-cac5c747ec69" (UID: "89b34d86-f828-4b79-ac76-cac5c747ec69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:36:09.409911 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.409652    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"]
Nov 04 00:36:09.499264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.499232    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.499264 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.499263    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.499501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.499278    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.499501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.499310    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmfs\" (UniqueName: \"kubernetes.io/projected/89b34d86-f828-4b79-ac76-cac5c747ec69-kube-api-access-7mmfs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.499501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.499325    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.499501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.499339    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b34d86-f828-4b79-ac76-cac5c747ec69-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.993379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.993342    5074 generic.go:356] "Generic (PLEG): container finished" podID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerID="caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060" exitCode=0
Nov 04 00:36:09.993539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.993388    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"
Nov 04 00:36:09.993539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.993426    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" event={"ID":"89b34d86-f828-4b79-ac76-cac5c747ec69","Type":"ContainerDied","Data":"caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060"}
Nov 04 00:36:09.993539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.993488    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54" event={"ID":"89b34d86-f828-4b79-ac76-cac5c747ec69","Type":"ContainerDied","Data":"acc83446fae10155347c6199824f30f94f04739fb90cb384d53376e6f2252298"}
Nov 04 00:36:09.993539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:09.993512    5074 scope.go:122] "RemoveContainer" containerID="caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060"
Nov 04 00:36:10.006811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:10.006791    5074 scope.go:122] "RemoveContainer" containerID="caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060"
Nov 04 00:36:10.007124 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:10.007106    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060\": container with ID starting with caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060 not found: ID does not exist" containerID="caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060"
Nov 04 00:36:10.007198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:10.007130    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060"} err="failed to get container status \"caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060\": rpc error: code = NotFound desc = could not find container \"caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060\": container with ID starting with caf545531fe28bd954f7c91643bda2ea07fbcf669edea3934c7da75a31ce4060 not found: ID does not exist"
Nov 04 00:36:10.026551 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:10.026521    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"]
Nov 04 00:36:10.043578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:10.043549    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-6nk54"]
Nov 04 00:36:11.019201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:11.019155    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" path="/var/lib/kubelet/pods/89b34d86-f828-4b79-ac76-cac5c747ec69/volumes"
Nov 04 00:36:12.258331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.258300    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 04 00:36:12.288613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.288590    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 04 00:36:12.366654 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.366621    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq\" (UniqueName: \"kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq\") pod \"d6def27d-ec93-4920-a8c6-0044521b9a03\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") "
Nov 04 00:36:12.366848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.366757    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume\") pod \"d6def27d-ec93-4920-a8c6-0044521b9a03\" (UID: \"d6def27d-ec93-4920-a8c6-0044521b9a03\") "
Nov 04 00:36:12.367198 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.367165    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume" pod "d6def27d-ec93-4920-a8c6-0044521b9a03" (UID: "d6def27d-ec93-4920-a8c6-0044521b9a03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:12.368947 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.368921    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq" pod "d6def27d-ec93-4920-a8c6-0044521b9a03" (UID: "d6def27d-ec93-4920-a8c6-0044521b9a03"). InnerVolumeSpecName "kube-api-access-nhckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:12.467703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.467627    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv\" (UniqueName: \"kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv\") pod \"d7e16c7d-f400-4b5a-9af4-6365060d9576\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") "
Nov 04 00:36:12.467915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.467763    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume\") pod \"d7e16c7d-f400-4b5a-9af4-6365060d9576\" (UID: \"d7e16c7d-f400-4b5a-9af4-6365060d9576\") "
Nov 04 00:36:12.468030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.468019    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6def27d-ec93-4920-a8c6-0044521b9a03-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:12.468073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.468032    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhckq\" (UniqueName: \"kubernetes.io/projected/d6def27d-ec93-4920-a8c6-0044521b9a03-kube-api-access-nhckq\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:12.468132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.468108    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume" pod "d7e16c7d-f400-4b5a-9af4-6365060d9576" (UID: "d7e16c7d-f400-4b5a-9af4-6365060d9576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:12.469753 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.469720    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv" pod "d7e16c7d-f400-4b5a-9af4-6365060d9576" (UID: "d7e16c7d-f400-4b5a-9af4-6365060d9576"). InnerVolumeSpecName "kube-api-access-xdnwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:12.569306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.569257    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e16c7d-f400-4b5a-9af4-6365060d9576-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:12.569306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:12.569303    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdnwv\" (UniqueName: \"kubernetes.io/projected/d7e16c7d-f400-4b5a-9af4-6365060d9576-kube-api-access-xdnwv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:13.020912 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.020869    5074 generic.go:356] "Generic (PLEG): container finished" podID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerID="544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f" exitCode=0
Nov 04 00:36:13.021092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.020917    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"
Nov 04 00:36:13.021092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.020945    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" event={"ID":"d6def27d-ec93-4920-a8c6-0044521b9a03","Type":"ContainerDied","Data":"544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f"}
Nov 04 00:36:13.021092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.021009    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22" event={"ID":"d6def27d-ec93-4920-a8c6-0044521b9a03","Type":"ContainerDied","Data":"a83f64a772998a4d5c1c3ebf829c595c8420f864ccb6df66d6e255bf831b1002"}
Nov 04 00:36:13.021092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.021044    5074 scope.go:122] "RemoveContainer" containerID="544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f"
Nov 04 00:36:13.023318 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.023290    5074 generic.go:356] "Generic (PLEG): container finished" podID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerID="ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915" exitCode=0
Nov 04 00:36:13.023434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.023385    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" event={"ID":"d7e16c7d-f400-4b5a-9af4-6365060d9576","Type":"ContainerDied","Data":"ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915"}
Nov 04 00:36:13.023434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.023422    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n" event={"ID":"d7e16c7d-f400-4b5a-9af4-6365060d9576","Type":"ContainerDied","Data":"379efbb57e611419b4a72568cde79feb52f9cf394c6493edbc51be54b9285491"}
Nov 04 00:36:13.023559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.023494    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"
Nov 04 00:36:13.035205 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.035186    5074 scope.go:122] "RemoveContainer" containerID="544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f"
Nov 04 00:36:13.036156 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:13.035715    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f\": container with ID starting with 544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f not found: ID does not exist" containerID="544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f"
Nov 04 00:36:13.036156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.035786    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f"} err="failed to get container status \"544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f\": rpc error: code = NotFound desc = could not find container \"544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f\": container with ID starting with 544dfb09bf62416f45569e6fd949012212b3f99e10f44972971bee13685b6c4f not found: ID does not exist"
Nov 04 00:36:13.036156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.035804    5074 scope.go:122] "RemoveContainer" containerID="ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915"
Nov 04 00:36:13.047431 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.047409    5074 scope.go:122] "RemoveContainer" containerID="ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915"
Nov 04 00:36:13.047759 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:13.047723    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915\": container with ID starting with ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915 not found: ID does not exist" containerID="ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915"
Nov 04 00:36:13.047877 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.047766    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915"} err="failed to get container status \"ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915\": rpc error: code = NotFound desc = could not find container \"ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915\": container with ID starting with ab4f7de40fd8fe0519c17521aedca721fbf8e656eec314e2ffa19e29c41cb915 not found: ID does not exist"
Nov 04 00:36:13.058832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.058810    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"]
Nov 04 00:36:13.063941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.063920    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8093f5-647fc7b7dd-92h22"]
Nov 04 00:36:13.076277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.076251    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"]
Nov 04 00:36:13.083495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:13.083474    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8093f5-68c664fff-tn62n"]
Nov 04 00:36:15.019594 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:15.019554    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" path="/var/lib/kubelet/pods/d6def27d-ec93-4920-a8c6-0044521b9a03/volumes"
Nov 04 00:36:15.020212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:15.020189    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" path="/var/lib/kubelet/pods/d7e16c7d-f400-4b5a-9af4-6365060d9576/volumes"
Nov 04 00:36:15.334870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:15.334780    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"]
Nov 04 00:36:15.342700 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:15.342668    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"]
Nov 04 00:36:20.335332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.335269    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd" podUID="dbde246c-8dc0-455f-bcf9-946d19291496" containerName="payload-processing" containerID="cri-o://79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83" gracePeriod=25
Nov 04 00:36:20.343992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.343919    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerName="payload-pre-processing" containerID="cri-o://6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8" gracePeriod=25
Nov 04 00:36:20.615145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.615116    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:36:20.662365 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.662323    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume\") pod \"dbde246c-8dc0-455f-bcf9-946d19291496\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") "
Nov 04 00:36:20.662647 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.662495    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8\" (UniqueName: \"kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8\") pod \"dbde246c-8dc0-455f-bcf9-946d19291496\" (UID: \"dbde246c-8dc0-455f-bcf9-946d19291496\") "
Nov 04 00:36:20.664188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.664155    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume" pod "dbde246c-8dc0-455f-bcf9-946d19291496" (UID: "dbde246c-8dc0-455f-bcf9-946d19291496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:20.671307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.671265    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8" pod "dbde246c-8dc0-455f-bcf9-946d19291496" (UID: "dbde246c-8dc0-455f-bcf9-946d19291496"). InnerVolumeSpecName "kube-api-access-5cls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:20.764540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.764497    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbde246c-8dc0-455f-bcf9-946d19291496-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:20.764540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.764536    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cls8\" (UniqueName: \"kubernetes.io/projected/dbde246c-8dc0-455f-bcf9-946d19291496-kube-api-access-5cls8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:20.938330 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:20.937464    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr" podUID="7bb3b247-d681-4d72-827a-0194a17f45af" containerName="main" containerID="cri-o://c38f7ec7ffb3def2f5dab70e1c9e76b5df520febf05dfa3f7710e38493e046af" gracePeriod=45
Nov 04 00:36:21.091782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.087517    5074 generic.go:356] "Generic (PLEG): container finished" podID="dbde246c-8dc0-455f-bcf9-946d19291496" containerID="79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83" exitCode=0
Nov 04 00:36:21.091782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.087612    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd" event={"ID":"dbde246c-8dc0-455f-bcf9-946d19291496","Type":"ContainerDied","Data":"79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83"}
Nov 04 00:36:21.091782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.087650    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd" event={"ID":"dbde246c-8dc0-455f-bcf9-946d19291496","Type":"ContainerDied","Data":"e47b46ef0a364355bd4476de518740bdc479b164e6208af58d0a5f30d4e51455"}
Nov 04 00:36:21.091782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.087670    5074 scope.go:122] "RemoveContainer" containerID="79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83"
Nov 04 00:36:21.091782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.087823    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"
Nov 04 00:36:21.092287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.091917    5074 generic.go:356] "Generic (PLEG): container finished" podID="7bb3b247-d681-4d72-827a-0194a17f45af" containerID="c38f7ec7ffb3def2f5dab70e1c9e76b5df520febf05dfa3f7710e38493e046af" exitCode=0
Nov 04 00:36:21.092287 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.091984    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr" event={"ID":"7bb3b247-d681-4d72-827a-0194a17f45af","Type":"ContainerDied","Data":"c38f7ec7ffb3def2f5dab70e1c9e76b5df520febf05dfa3f7710e38493e046af"}
Nov 04 00:36:21.102334 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.102300    5074 scope.go:122] "RemoveContainer" containerID="79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83"
Nov 04 00:36:21.102621 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:21.102589    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83\": container with ID starting with 79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83 not found: ID does not exist" containerID="79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83"
Nov 04 00:36:21.102703 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.102630    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83"} err="failed to get container status \"79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83\": rpc error: code = NotFound desc = could not find container \"79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83\": container with ID starting with 79fc9aacad946997f692927db521a187e2d5663085ba766f346004abdba49f83 not found: ID does not exist"
Nov 04 00:36:21.114218 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.114121    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"]
Nov 04 00:36:21.118489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.118461    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8895cb-547bb76cf8-s68zd"]
Nov 04 00:36:21.238615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.238582    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:36:21.269501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269461    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r\" (UniqueName: \"kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r\") pod \"7bb3b247-d681-4d72-827a-0194a17f45af\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") "
Nov 04 00:36:21.269695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269526    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache\") pod \"7bb3b247-d681-4d72-827a-0194a17f45af\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") "
Nov 04 00:36:21.269695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269579    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs\") pod \"7bb3b247-d681-4d72-827a-0194a17f45af\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") "
Nov 04 00:36:21.269695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269608    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm\") pod \"7bb3b247-d681-4d72-827a-0194a17f45af\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") "
Nov 04 00:36:21.269695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269631    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir\") pod \"7bb3b247-d681-4d72-827a-0194a17f45af\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") "
Nov 04 00:36:21.269695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269681    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home\") pod \"7bb3b247-d681-4d72-827a-0194a17f45af\" (UID: \"7bb3b247-d681-4d72-827a-0194a17f45af\") "
Nov 04 00:36:21.270046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.269875    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache" pod "7bb3b247-d681-4d72-827a-0194a17f45af" (UID: "7bb3b247-d681-4d72-827a-0194a17f45af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:21.270115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.270073    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home" pod "7bb3b247-d681-4d72-827a-0194a17f45af" (UID: "7bb3b247-d681-4d72-827a-0194a17f45af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:21.270239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.270205    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir" pod "7bb3b247-d681-4d72-827a-0194a17f45af" (UID: "7bb3b247-d681-4d72-827a-0194a17f45af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:21.270306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.270231    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:21.270306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.270256    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:21.271861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.271833    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm" pod "7bb3b247-d681-4d72-827a-0194a17f45af" (UID: "7bb3b247-d681-4d72-827a-0194a17f45af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:21.271861 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.271852    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs" pod "7bb3b247-d681-4d72-827a-0194a17f45af" (UID: "7bb3b247-d681-4d72-827a-0194a17f45af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:21.272144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.272123    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r" pod "7bb3b247-d681-4d72-827a-0194a17f45af" (UID: "7bb3b247-d681-4d72-827a-0194a17f45af"). InnerVolumeSpecName "kube-api-access-g9v9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:21.372614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.372577    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb3b247-d681-4d72-827a-0194a17f45af-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:21.372614 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.372615    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:21.373283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.372630    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb3b247-d681-4d72-827a-0194a17f45af-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:21.373283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:21.372645    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9v9r\" (UniqueName: \"kubernetes.io/projected/7bb3b247-d681-4d72-827a-0194a17f45af-kube-api-access-g9v9r\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:22.100996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:22.100952    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"
Nov 04 00:36:22.100996 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:22.100976    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr" event={"ID":"7bb3b247-d681-4d72-827a-0194a17f45af","Type":"ContainerDied","Data":"46c0a039dc1ca3ecdf375c3b842d47f9f93a4241462b2cb8a826a1ab31ab72ea"}
Nov 04 00:36:22.101288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:22.101035    5074 scope.go:122] "RemoveContainer" containerID="c38f7ec7ffb3def2f5dab70e1c9e76b5df520febf05dfa3f7710e38493e046af"
Nov 04 00:36:22.139484 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:22.139447    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"]
Nov 04 00:36:22.142629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:22.142605    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b3bea7/sub-test-model-b3bea7-kserve-6b5949bf97-px7pr"]
Nov 04 00:36:23.018486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:23.018438    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb3b247-d681-4d72-827a-0194a17f45af" path="/var/lib/kubelet/pods/7bb3b247-d681-4d72-827a-0194a17f45af/volumes"
Nov 04 00:36:23.019057 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:23.019034    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbde246c-8dc0-455f-bcf9-946d19291496" path="/var/lib/kubelet/pods/dbde246c-8dc0-455f-bcf9-946d19291496/volumes"
Nov 04 00:36:24.185384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.185350    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" podUID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" containerName="main" containerID="cri-o://f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a" gracePeriod=45
Nov 04 00:36:24.223917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.223883    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:24.435344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.435305    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" containerID="cri-o://99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5" gracePeriod=45
Nov 04 00:36:24.568301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.568277    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:36:24.611042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.610987    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused"
Nov 04 00:36:24.611281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611214    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache\") pod \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") "
Nov 04 00:36:24.611433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611317    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home\") pod \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") "
Nov 04 00:36:24.611433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611363    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm\") pod \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") "
Nov 04 00:36:24.611433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611394    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir\") pod \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") "
Nov 04 00:36:24.611433 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611424    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs\") pod \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") "
Nov 04 00:36:24.611691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611513    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj\" (UniqueName: \"kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj\") pod \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\" (UID: \"811cb4e3-12f4-4791-9ed1-0807d4045f5e\") "
Nov 04 00:36:24.611691 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611534    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache" pod "811cb4e3-12f4-4791-9ed1-0807d4045f5e" (UID: "811cb4e3-12f4-4791-9ed1-0807d4045f5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.611965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.611942    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.612540 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.612496    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir" pod "811cb4e3-12f4-4791-9ed1-0807d4045f5e" (UID: "811cb4e3-12f4-4791-9ed1-0807d4045f5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.612674 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.612650    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home" pod "811cb4e3-12f4-4791-9ed1-0807d4045f5e" (UID: "811cb4e3-12f4-4791-9ed1-0807d4045f5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.616186 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.615909    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj" pod "811cb4e3-12f4-4791-9ed1-0807d4045f5e" (UID: "811cb4e3-12f4-4791-9ed1-0807d4045f5e"). InnerVolumeSpecName "kube-api-access-vjvwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:24.617905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.617881    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs" pod "811cb4e3-12f4-4791-9ed1-0807d4045f5e" (UID: "811cb4e3-12f4-4791-9ed1-0807d4045f5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:24.618826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.618804    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm" pod "811cb4e3-12f4-4791-9ed1-0807d4045f5e" (UID: "811cb4e3-12f4-4791-9ed1-0807d4045f5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.715552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.715457    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.715552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.715502    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.715552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.715515    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.715552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.715528    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811cb4e3-12f4-4791-9ed1-0807d4045f5e-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.715552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.715545    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvwj\" (UniqueName: \"kubernetes.io/projected/811cb4e3-12f4-4791-9ed1-0807d4045f5e-kube-api-access-vjvwj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.737336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.737250    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 04 00:36:24.816270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816236    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs\") pod \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") "
Nov 04 00:36:24.816442 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816356    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm\") pod \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") "
Nov 04 00:36:24.816442 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816410    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx\" (UniqueName: \"kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx\") pod \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") "
Nov 04 00:36:24.816585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816446    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir\") pod \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") "
Nov 04 00:36:24.816585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816511    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home\") pod \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") "
Nov 04 00:36:24.816585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816552    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache\") pod \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\" (UID: \"73141bd1-58b2-4f0c-9fda-b395ca8c81a6\") "
Nov 04 00:36:24.817009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.816976    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir" pod "73141bd1-58b2-4f0c-9fda-b395ca8c81a6" (UID: "73141bd1-58b2-4f0c-9fda-b395ca8c81a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.817170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.817148    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache" pod "73141bd1-58b2-4f0c-9fda-b395ca8c81a6" (UID: "73141bd1-58b2-4f0c-9fda-b395ca8c81a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.818329 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.817171    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home" pod "73141bd1-58b2-4f0c-9fda-b395ca8c81a6" (UID: "73141bd1-58b2-4f0c-9fda-b395ca8c81a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.821977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.821947    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx" pod "73141bd1-58b2-4f0c-9fda-b395ca8c81a6" (UID: "73141bd1-58b2-4f0c-9fda-b395ca8c81a6"). InnerVolumeSpecName "kube-api-access-qk7mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:24.822107 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.822041    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs" pod "73141bd1-58b2-4f0c-9fda-b395ca8c81a6" (UID: "73141bd1-58b2-4f0c-9fda-b395ca8c81a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:24.822561 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.822539    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm" pod "73141bd1-58b2-4f0c-9fda-b395ca8c81a6" (UID: "73141bd1-58b2-4f0c-9fda-b395ca8c81a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.918384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.918345    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.918384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.918383    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk7mx\" (UniqueName: \"kubernetes.io/projected/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-kube-api-access-qk7mx\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.918626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.918398    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.918626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.918411    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.918626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.918424    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.918626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:24.918435    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73141bd1-58b2-4f0c-9fda-b395ca8c81a6-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:25.127668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.127632    5074 generic.go:356] "Generic (PLEG): container finished" podID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" containerID="f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a" exitCode=0
Nov 04 00:36:25.127902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.127696    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"
Nov 04 00:36:25.127902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.127715    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" event={"ID":"811cb4e3-12f4-4791-9ed1-0807d4045f5e","Type":"ContainerDied","Data":"f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a"}
Nov 04 00:36:25.127902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.127777    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" event={"ID":"811cb4e3-12f4-4791-9ed1-0807d4045f5e","Type":"ContainerDied","Data":"63a66b55bdefccc25c52a4ceb4d5eee77f992a69450e5a077ccda68bd6beafdf"}
Nov 04 00:36:25.127902 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.127800    5074 scope.go:122] "RemoveContainer" containerID="f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a"
Nov 04 00:36:25.129976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.129950    5074 generic.go:356] "Generic (PLEG): container finished" podID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerID="99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5" exitCode=0
Nov 04 00:36:25.130208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.130001    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" event={"ID":"73141bd1-58b2-4f0c-9fda-b395ca8c81a6","Type":"ContainerDied","Data":"99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5"}
Nov 04 00:36:25.130208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.130037    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4" event={"ID":"73141bd1-58b2-4f0c-9fda-b395ca8c81a6","Type":"ContainerDied","Data":"99c2415df60bc66b2b2c674a4381b62e4780647afe56c7f0d87e2c065ad4c9f5"}
Nov 04 00:36:25.130208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.130096    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"
Nov 04 00:36:25.138354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.138335    5074 scope.go:122] "RemoveContainer" containerID="f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a"
Nov 04 00:36:25.138671 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:25.138649    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a\": container with ID starting with f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a not found: ID does not exist" containerID="f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a"
Nov 04 00:36:25.138789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.138680    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a"} err="failed to get container status \"f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a\": rpc error: code = NotFound desc = could not find container \"f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a\": container with ID starting with f607a4da293c36a1f72320e8a445d0874bd42624f825829107b919c1a8d5ed5a not found: ID does not exist"
Nov 04 00:36:25.138789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.138702    5074 scope.go:122] "RemoveContainer" containerID="99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5"
Nov 04 00:36:25.148905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.148883    5074 scope.go:122] "RemoveContainer" containerID="99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5"
Nov 04 00:36:25.149191 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:25.149170    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5\": container with ID starting with 99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5 not found: ID does not exist" containerID="99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5"
Nov 04 00:36:25.149272 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.149201    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5"} err="failed to get container status \"99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5\": rpc error: code = NotFound desc = could not find container \"99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5\": container with ID starting with 99f75ffde388327c15afdf06be2b636cdc785a0176230e2b29d38ed2d0ca21b5 not found: ID does not exist"
Nov 04 00:36:25.173847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.173819    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"]
Nov 04 00:36:25.192083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.192049    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv"]
Nov 04 00:36:25.248111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.237514    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"]
Nov 04 00:36:25.248111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.246489    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/premium-w2-5738a9-kserve-6f6f568887-8vdq4"]
Nov 04 00:36:25.367413 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:25.367319    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-15b02a/sub-test-model-15b02a-kserve-7cd9b5647f-h9frv" podUID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" containerName="main" probeResult="failure" output="Get \"https://10.132.1.160:8000/ready\": context deadline exceeded"
Nov 04 00:36:26.197988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.197945    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"]
Nov 04 00:36:26.223020 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.209317    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"]
Nov 04 00:36:26.845926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.845892    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"]
Nov 04 00:36:26.846412 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846397    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerName="payload-processing"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846414    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerName="payload-processing"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846425    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerName="payload-pre-processing"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846430    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerName="payload-pre-processing"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846440    5074 cpu_manager.go:414] "containerMap: removing container" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846445    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846452    5074 cpu_manager.go:414] "containerMap: removing container" podUID="dbde246c-8dc0-455f-bcf9-946d19291496" containerName="payload-processing"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846456    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbde246c-8dc0-455f-bcf9-946d19291496" containerName="payload-processing"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846463    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9d9eaedd-71e2-41a1-b61f-61c385c290c7" containerName="revoke-keys"
Nov 04 00:36:26.846467 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846467    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9eaedd-71e2-41a1-b61f-61c385c290c7" containerName="revoke-keys"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846477    5074 cpu_manager.go:414] "containerMap: removing container" podUID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" containerName="main"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846482    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" containerName="main"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846494    5074 cpu_manager.go:414] "containerMap: removing container" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846499    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846521    5074 cpu_manager.go:414] "containerMap: removing container" podUID="7bb3b247-d681-4d72-827a-0194a17f45af" containerName="main"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846527    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb3b247-d681-4d72-827a-0194a17f45af" containerName="main"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846594    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b34d86-f828-4b79-ac76-cac5c747ec69" containerName="istio-proxy"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846606    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb3b247-d681-4d72-827a-0194a17f45af" containerName="main"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846615    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6def27d-ec93-4920-a8c6-0044521b9a03" containerName="payload-processing"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846621    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e16c7d-f400-4b5a-9af4-6365060d9576" containerName="payload-pre-processing"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846629    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" containerName="main"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846635    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbde246c-8dc0-455f-bcf9-946d19291496" containerName="payload-processing"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846641    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9eaedd-71e2-41a1-b61f-61c385c290c7" containerName="revoke-keys"
Nov 04 00:36:26.846780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.846647    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" containerName="main"
Nov 04 00:36:26.851370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.851344    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:26.857707 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.857682    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:36:26.857908 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.857786    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:36:26.860581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.860379    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"]
Nov 04 00:36:26.943993 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.943958    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gsw\" (UniqueName: \"kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:26.944177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.944002    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:26.944227 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:26.944192    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.017793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.017756    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73141bd1-58b2-4f0c-9fda-b395ca8c81a6" path="/var/lib/kubelet/pods/73141bd1-58b2-4f0c-9fda-b395ca8c81a6/volumes"
Nov 04 00:36:27.018252 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.018236    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811cb4e3-12f4-4791-9ed1-0807d4045f5e" path="/var/lib/kubelet/pods/811cb4e3-12f4-4791-9ed1-0807d4045f5e/volumes"
Nov 04 00:36:27.044975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.044944    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gsw\" (UniqueName: \"kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.044975 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.044978    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.045177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.045051    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.045816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.045794    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.047548 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.047527    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.067655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.067630    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gsw\" (UniqueName: \"kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw\") pod \"maas-api-revoke-sub-3d1421d2-72qmm\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.169555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.169462    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:27.459168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:27.459132    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"]
Nov 04 00:36:27.461130 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:36:27.461103    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221cf970_43a4_4aca_bef3_d3efe4d53398.slice/crio-58de3498f6dbe3a16d4224d0a6f1cfd50bb0ab057964a53595e7f296a916e40a WatchSource:0}: Error finding container 58de3498f6dbe3a16d4224d0a6f1cfd50bb0ab057964a53595e7f296a916e40a: Status 404 returned error can't find the container with id 58de3498f6dbe3a16d4224d0a6f1cfd50bb0ab057964a53595e7f296a916e40a
Nov 04 00:36:28.025512 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:28.025401    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:36:28.157408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:28.157365    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm" event={"ID":"221cf970-43a4-4aca-bef3-d3efe4d53398","Type":"ContainerStarted","Data":"fb7a67d19b11e5ff83dc1702d01d708d6547404de5912a58ce62c20001e9cf4d"}
Nov 04 00:36:28.157573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:28.157431    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm" event={"ID":"221cf970-43a4-4aca-bef3-d3efe4d53398","Type":"ContainerStarted","Data":"58de3498f6dbe3a16d4224d0a6f1cfd50bb0ab057964a53595e7f296a916e40a"}
Nov 04 00:36:28.181089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:28.181033    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm" podStartSLOduration=2.181016173 podStartE2EDuration="2.181016173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:36:28.178443436 +0000 UTC m=+4053.278043581" watchObservedRunningTime="2026-11-04 00:36:28.181016173 +0000 UTC m=+4053.280616319"
Nov 04 00:36:29.165503 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:29.165461    5074 generic.go:356] "Generic (PLEG): container finished" podID="221cf970-43a4-4aca-bef3-d3efe4d53398" containerID="fb7a67d19b11e5ff83dc1702d01d708d6547404de5912a58ce62c20001e9cf4d" exitCode=0
Nov 04 00:36:29.165977 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:29.165549    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm" event={"ID":"221cf970-43a4-4aca-bef3-d3efe4d53398","Type":"ContainerDied","Data":"fb7a67d19b11e5ff83dc1702d01d708d6547404de5912a58ce62c20001e9cf4d"}
Nov 04 00:36:30.323211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.323178    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:30.378224 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.378185    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca\") pod \"221cf970-43a4-4aca-bef3-d3efe4d53398\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") "
Nov 04 00:36:30.378370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.378248    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw\" (UniqueName: \"kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw\") pod \"221cf970-43a4-4aca-bef3-d3efe4d53398\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") "
Nov 04 00:36:30.378370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.378343    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca\") pod \"221cf970-43a4-4aca-bef3-d3efe4d53398\" (UID: \"221cf970-43a4-4aca-bef3-d3efe4d53398\") "
Nov 04 00:36:30.378624 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.378599    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca" pod "221cf970-43a4-4aca-bef3-d3efe4d53398" (UID: "221cf970-43a4-4aca-bef3-d3efe4d53398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:30.378711 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.378691    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:30.380610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.380583    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca" pod "221cf970-43a4-4aca-bef3-d3efe4d53398" (UID: "221cf970-43a4-4aca-bef3-d3efe4d53398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:30.380723 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.380702    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw" pod "221cf970-43a4-4aca-bef3-d3efe4d53398" (UID: "221cf970-43a4-4aca-bef3-d3efe4d53398"). InnerVolumeSpecName "kube-api-access-p8gsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:30.479295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.479261    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gsw\" (UniqueName: \"kubernetes.io/projected/221cf970-43a4-4aca-bef3-d3efe4d53398-kube-api-access-p8gsw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:30.479295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:30.479292    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221cf970-43a4-4aca-bef3-d3efe4d53398-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:31.184277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.184208    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm" event={"ID":"221cf970-43a4-4aca-bef3-d3efe4d53398","Type":"ContainerDied","Data":"58de3498f6dbe3a16d4224d0a6f1cfd50bb0ab057964a53595e7f296a916e40a"}
Nov 04 00:36:31.184277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.184280    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58de3498f6dbe3a16d4224d0a6f1cfd50bb0ab057964a53595e7f296a916e40a"
Nov 04 00:36:31.184536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.184363    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"
Nov 04 00:36:31.198396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.198360    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4" podUID="a47312f3-f293-41d2-b581-a77372061952" containerName="payload-processing" containerID="cri-o://c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79" gracePeriod=25
Nov 04 00:36:31.210669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.210632    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4" podUID="b2d49ec7-a482-4f38-b933-b057ae376e4f" containerName="payload-pre-processing" containerID="cri-o://b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd" gracePeriod=25
Nov 04 00:36:31.486683 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.486657    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 04 00:36:31.536166 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.536127    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 04 00:36:31.590453 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.590364    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume\") pod \"a47312f3-f293-41d2-b581-a77372061952\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") "
Nov 04 00:36:31.590615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.590480    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k\" (UniqueName: \"kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k\") pod \"a47312f3-f293-41d2-b581-a77372061952\" (UID: \"a47312f3-f293-41d2-b581-a77372061952\") "
Nov 04 00:36:31.590615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.590542    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn\" (UniqueName: \"kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn\") pod \"b2d49ec7-a482-4f38-b933-b057ae376e4f\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") "
Nov 04 00:36:31.590615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.590584    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume\") pod \"b2d49ec7-a482-4f38-b933-b057ae376e4f\" (UID: \"b2d49ec7-a482-4f38-b933-b057ae376e4f\") "
Nov 04 00:36:31.590846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.590822    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume" pod "a47312f3-f293-41d2-b581-a77372061952" (UID: "a47312f3-f293-41d2-b581-a77372061952"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:31.591175 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.591151    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47312f3-f293-41d2-b581-a77372061952-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:31.591284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.591194    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume" pod "b2d49ec7-a482-4f38-b933-b057ae376e4f" (UID: "b2d49ec7-a482-4f38-b933-b057ae376e4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:31.593118 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.593092    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn" pod "b2d49ec7-a482-4f38-b933-b057ae376e4f" (UID: "b2d49ec7-a482-4f38-b933-b057ae376e4f"). InnerVolumeSpecName "kube-api-access-m4jhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:31.593255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.593145    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k" pod "a47312f3-f293-41d2-b581-a77372061952" (UID: "a47312f3-f293-41d2-b581-a77372061952"). InnerVolumeSpecName "kube-api-access-qmq5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:31.692273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.692235    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4jhn\" (UniqueName: \"kubernetes.io/projected/b2d49ec7-a482-4f38-b933-b057ae376e4f-kube-api-access-m4jhn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:31.692273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.692267    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d49ec7-a482-4f38-b933-b057ae376e4f-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:31.692273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:31.692277    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmq5k\" (UniqueName: \"kubernetes.io/projected/a47312f3-f293-41d2-b581-a77372061952-kube-api-access-qmq5k\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:32.110762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.110695    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"]
Nov 04 00:36:32.111486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111456    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b2d49ec7-a482-4f38-b933-b057ae376e4f" containerName="payload-pre-processing"
Nov 04 00:36:32.111486 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111483    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d49ec7-a482-4f38-b933-b057ae376e4f" containerName="payload-pre-processing"
Nov 04 00:36:32.111651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111517    5074 cpu_manager.go:414] "containerMap: removing container" podUID="221cf970-43a4-4aca-bef3-d3efe4d53398" containerName="revoke-keys"
Nov 04 00:36:32.111651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111525    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221cf970-43a4-4aca-bef3-d3efe4d53398" containerName="revoke-keys"
Nov 04 00:36:32.111651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111552    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a47312f3-f293-41d2-b581-a77372061952" containerName="payload-processing"
Nov 04 00:36:32.111651 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111560    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47312f3-f293-41d2-b581-a77372061952" containerName="payload-processing"
Nov 04 00:36:32.111818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111697    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d49ec7-a482-4f38-b933-b057ae376e4f" containerName="payload-pre-processing"
Nov 04 00:36:32.111818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111712    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="221cf970-43a4-4aca-bef3-d3efe4d53398" containerName="revoke-keys"
Nov 04 00:36:32.111818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.111724    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47312f3-f293-41d2-b581-a77372061952" containerName="payload-processing"
Nov 04 00:36:32.115801 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.115781    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.122879 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.122849    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:36:32.123119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.123096    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"]
Nov 04 00:36:32.124649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.124625    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:36:32.193863 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.193829    5074 generic.go:356] "Generic (PLEG): container finished" podID="a47312f3-f293-41d2-b581-a77372061952" containerID="c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79" exitCode=0
Nov 04 00:36:32.194035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.193881    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"
Nov 04 00:36:32.194035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.193909    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4" event={"ID":"a47312f3-f293-41d2-b581-a77372061952","Type":"ContainerDied","Data":"c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79"}
Nov 04 00:36:32.194035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.193976    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4" event={"ID":"a47312f3-f293-41d2-b581-a77372061952","Type":"ContainerDied","Data":"1c5cc14a8787dd6ce27da99727849b19eeb78f07bc06d6105cfdfd9b88dc89f0"}
Nov 04 00:36:32.194035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.194003    5074 scope.go:122] "RemoveContainer" containerID="c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79"
Nov 04 00:36:32.195574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.195555    5074 generic.go:356] "Generic (PLEG): container finished" podID="b2d49ec7-a482-4f38-b933-b057ae376e4f" containerID="b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd" exitCode=0
Nov 04 00:36:32.195663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.195605    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4" event={"ID":"b2d49ec7-a482-4f38-b933-b057ae376e4f","Type":"ContainerDied","Data":"b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd"}
Nov 04 00:36:32.195663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.195630    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"
Nov 04 00:36:32.195663 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.195638    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4" event={"ID":"b2d49ec7-a482-4f38-b933-b057ae376e4f","Type":"ContainerDied","Data":"c0d0c1847f86462b62fd7d6d8304d184dfcc88303c5bfb53f486ffcd6301c926"}
Nov 04 00:36:32.196452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.196426    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.196619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.196592    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvsz\" (UniqueName: \"kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.196687 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.196672    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.207828 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.207511    5074 scope.go:122] "RemoveContainer" containerID="c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79"
Nov 04 00:36:32.207965 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:32.207932    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79\": container with ID starting with c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79 not found: ID does not exist" containerID="c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79"
Nov 04 00:36:32.208043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.207974    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79"} err="failed to get container status \"c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79\": rpc error: code = NotFound desc = could not find container \"c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79\": container with ID starting with c6a13358642b6a6f65b4739cb2161b0c853072e0a3590af26835383252be9e79 not found: ID does not exist"
Nov 04 00:36:32.208043 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.207996    5074 scope.go:122] "RemoveContainer" containerID="b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd"
Nov 04 00:36:32.217765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.217729    5074 scope.go:122] "RemoveContainer" containerID="b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd"
Nov 04 00:36:32.218027 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:32.217996    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd\": container with ID starting with b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd not found: ID does not exist" containerID="b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd"
Nov 04 00:36:32.218094 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.218031    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd"} err="failed to get container status \"b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd\": rpc error: code = NotFound desc = could not find container \"b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd\": container with ID starting with b6bc15b82b007f2550546050298a35fe225911cfa612b09810adc27e76394afd not found: ID does not exist"
Nov 04 00:36:32.226675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.226653    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"]
Nov 04 00:36:32.231676 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.231650    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5738a9-6cffdb57c6-hf4w4"]
Nov 04 00:36:32.241958 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.241933    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"]
Nov 04 00:36:32.246578 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.246557    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5738a9-567b7cf8c5-zjfm4"]
Nov 04 00:36:32.297372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.297340    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.297516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.297451    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvsz\" (UniqueName: \"kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.297516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.297511    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.298362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.298337    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.300176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.300152    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.310223 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.310202    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvsz\" (UniqueName: \"kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz\") pod \"maas-api-revoke-sub-9f906e62-6lmtw\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.433188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.433157    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"]
Nov 04 00:36:32.435058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.435030    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:32.764531 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:32.764494    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"]
Nov 04 00:36:32.765247 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:36:32.765218    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc07ee9_08a5_4d26_81af_dd06828fe786.slice/crio-a54639aa1e11115f74cf9700faa7b0172942a500c68a758bffc826c26573cbf3 WatchSource:0}: Error finding container a54639aa1e11115f74cf9700faa7b0172942a500c68a758bffc826c26573cbf3: Status 404 returned error can't find the container with id a54639aa1e11115f74cf9700faa7b0172942a500c68a758bffc826c26573cbf3
Nov 04 00:36:33.018087 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:33.018052    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47312f3-f293-41d2-b581-a77372061952" path="/var/lib/kubelet/pods/a47312f3-f293-41d2-b581-a77372061952/volumes"
Nov 04 00:36:33.018564 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:33.018544    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d49ec7-a482-4f38-b933-b057ae376e4f" path="/var/lib/kubelet/pods/b2d49ec7-a482-4f38-b933-b057ae376e4f/volumes"
Nov 04 00:36:33.207714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:33.207677    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw" event={"ID":"8bc07ee9-08a5-4d26-81af-dd06828fe786","Type":"ContainerStarted","Data":"d788af6c6f696b29324a21c8984af9a8c2e54b3ffdba40744efe5c3c551d6d2e"}
Nov 04 00:36:33.207970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:33.207786    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw" event={"ID":"8bc07ee9-08a5-4d26-81af-dd06828fe786","Type":"ContainerStarted","Data":"a54639aa1e11115f74cf9700faa7b0172942a500c68a758bffc826c26573cbf3"}
Nov 04 00:36:33.230295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:33.230249    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw" podStartSLOduration=1.230235094 podStartE2EDuration="1.230235094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:36:33.226869987 +0000 UTC m=+4058.326470136" watchObservedRunningTime="2026-11-04 00:36:33.230235094 +0000 UTC m=+4058.329835271"
Nov 04 00:36:34.217838 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:34.217796    5074 generic.go:356] "Generic (PLEG): container finished" podID="8bc07ee9-08a5-4d26-81af-dd06828fe786" containerID="d788af6c6f696b29324a21c8984af9a8c2e54b3ffdba40744efe5c3c551d6d2e" exitCode=0
Nov 04 00:36:34.218262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:34.217883    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw" event={"ID":"8bc07ee9-08a5-4d26-81af-dd06828fe786","Type":"ContainerDied","Data":"d788af6c6f696b29324a21c8984af9a8c2e54b3ffdba40744efe5c3c551d6d2e"}
Nov 04 00:36:34.223040 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:34.223002    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:35.380982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.380956    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:35.432342 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.432303    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca\") pod \"8bc07ee9-08a5-4d26-81af-dd06828fe786\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") "
Nov 04 00:36:35.432537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.432381    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca\") pod \"8bc07ee9-08a5-4d26-81af-dd06828fe786\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") "
Nov 04 00:36:35.432537 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.432508    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz\" (UniqueName: \"kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz\") pod \"8bc07ee9-08a5-4d26-81af-dd06828fe786\" (UID: \"8bc07ee9-08a5-4d26-81af-dd06828fe786\") "
Nov 04 00:36:35.433131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.432842    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca" pod "8bc07ee9-08a5-4d26-81af-dd06828fe786" (UID: "8bc07ee9-08a5-4d26-81af-dd06828fe786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:35.433131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.432975    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:35.435114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.435086    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca" pod "8bc07ee9-08a5-4d26-81af-dd06828fe786" (UID: "8bc07ee9-08a5-4d26-81af-dd06828fe786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:35.440181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.440149    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz" pod "8bc07ee9-08a5-4d26-81af-dd06828fe786" (UID: "8bc07ee9-08a5-4d26-81af-dd06828fe786"). InnerVolumeSpecName "kube-api-access-zxvsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:35.534381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.534311    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvsz\" (UniqueName: \"kubernetes.io/projected/8bc07ee9-08a5-4d26-81af-dd06828fe786-kube-api-access-zxvsz\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:35.534381 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:35.534345    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc07ee9-08a5-4d26-81af-dd06828fe786-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:36.233717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:36.233676    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw" event={"ID":"8bc07ee9-08a5-4d26-81af-dd06828fe786","Type":"ContainerDied","Data":"a54639aa1e11115f74cf9700faa7b0172942a500c68a758bffc826c26573cbf3"}
Nov 04 00:36:36.233974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:36.233733    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a54639aa1e11115f74cf9700faa7b0172942a500c68a758bffc826c26573cbf3"
Nov 04 00:36:36.233974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:36.233822    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"
Nov 04 00:36:36.700766 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:36.700716    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"]
Nov 04 00:36:37.434062 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:37.434005    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" podUID="07237989-93da-496c-8fe0-60e681a60ded" containerName="payload-pre-processing" containerID="cri-o://463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd" gracePeriod=25
Nov 04 00:36:40.617625 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.617553    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:36:40.693144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.693110    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t\" (UniqueName: \"kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t\") pod \"c633171b-c6c0-4f2f-9723-b8aa527533df\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") "
Nov 04 00:36:40.693289 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.693156    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume\") pod \"c633171b-c6c0-4f2f-9723-b8aa527533df\" (UID: \"c633171b-c6c0-4f2f-9723-b8aa527533df\") "
Nov 04 00:36:40.693596 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.693569    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume" pod "c633171b-c6c0-4f2f-9723-b8aa527533df" (UID: "c633171b-c6c0-4f2f-9723-b8aa527533df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:40.695293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.695270    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t" pod "c633171b-c6c0-4f2f-9723-b8aa527533df" (UID: "c633171b-c6c0-4f2f-9723-b8aa527533df"). InnerVolumeSpecName "kube-api-access-95d6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:40.794483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.794452    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95d6t\" (UniqueName: \"kubernetes.io/projected/c633171b-c6c0-4f2f-9723-b8aa527533df-kube-api-access-95d6t\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:40.794483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:40.794480    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c633171b-c6c0-4f2f-9723-b8aa527533df-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:41.077680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.077630    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" podUID="07237989-93da-496c-8fe0-60e681a60ded" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:41.273155 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.273118    5074 generic.go:356] "Generic (PLEG): container finished" podID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerID="6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8" exitCode=0
Nov 04 00:36:41.273366 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.273167    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"
Nov 04 00:36:41.273366 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.273207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" event={"ID":"c633171b-c6c0-4f2f-9723-b8aa527533df","Type":"ContainerDied","Data":"6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8"}
Nov 04 00:36:41.273366 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.273266    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f" event={"ID":"c633171b-c6c0-4f2f-9723-b8aa527533df","Type":"ContainerDied","Data":"819759fe75810223edde88860125788475f69496ba02dfa89f1d018a2bacfc2f"}
Nov 04 00:36:41.273366 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.273295    5074 scope.go:122] "RemoveContainer" containerID="6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8"
Nov 04 00:36:41.283678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.283630    5074 scope.go:122] "RemoveContainer" containerID="6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8"
Nov 04 00:36:41.283984 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:41.283960    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8\": container with ID starting with 6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8 not found: ID does not exist" containerID="6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8"
Nov 04 00:36:41.284042 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.283994    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8"} err="failed to get container status \"6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8\": rpc error: code = NotFound desc = could not find container \"6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8\": container with ID starting with 6a2a49afafb9a68d1c7650bd383d0f39474c0c22128e3c4cd45b6cbb9e288cd8 not found: ID does not exist"
Nov 04 00:36:41.295574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.295548    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"]
Nov 04 00:36:41.299006 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:41.298982    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8895cb-678c987dc-phf6f"]
Nov 04 00:36:43.024715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:43.024664    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" path="/var/lib/kubelet/pods/c633171b-c6c0-4f2f-9723-b8aa527533df/volumes"
Nov 04 00:36:44.080090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080051    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"]
Nov 04 00:36:44.080598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080535    5074 cpu_manager.go:414] "containerMap: removing container" podUID="8bc07ee9-08a5-4d26-81af-dd06828fe786" containerName="revoke-keys"
Nov 04 00:36:44.080598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080547    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc07ee9-08a5-4d26-81af-dd06828fe786" containerName="revoke-keys"
Nov 04 00:36:44.080598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080561    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerName="payload-pre-processing"
Nov 04 00:36:44.080598 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080567    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerName="payload-pre-processing"
Nov 04 00:36:44.080878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080666    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc07ee9-08a5-4d26-81af-dd06828fe786" containerName="revoke-keys"
Nov 04 00:36:44.080878 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.080675    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c633171b-c6c0-4f2f-9723-b8aa527533df" containerName="payload-pre-processing"
Nov 04 00:36:44.085250 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.085225    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.095416 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.095386    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"]
Nov 04 00:36:44.132815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.132778    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx56m\" (UniqueName: \"kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.133003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.132904    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.133111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.133029    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.185215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.185182    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:36:44.185386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.185216    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:36:44.237964 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.237115    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.238339 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.238309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx56m\" (UniqueName: \"kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.238814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.238791    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.240259 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.240237    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.253007 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.252980    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.255439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.255408    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx56m\" (UniqueName: \"kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m\") pod \"maas-api-revoke-sub-c800c36a-lfzt5\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.401997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.401907    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:44.687300 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:44.687269    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"]
Nov 04 00:36:44.688598 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:36:44.688572    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e2500a_38b3_4493_8ce1_84f2c82c01d6.slice/crio-e644ecb2bd693c07cb7fcd925c6431844a66033c3dd9dbb32b5e8a9e41808534 WatchSource:0}: Error finding container e644ecb2bd693c07cb7fcd925c6431844a66033c3dd9dbb32b5e8a9e41808534: Status 404 returned error can't find the container with id e644ecb2bd693c07cb7fcd925c6431844a66033c3dd9dbb32b5e8a9e41808534
Nov 04 00:36:45.303994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:45.303884    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5" event={"ID":"13e2500a-38b3-4493-8ce1-84f2c82c01d6","Type":"ContainerStarted","Data":"8b14895b44852180aef7688f2816f86a15d3d2a9eb99f99a1468d31a3bc36e68"}
Nov 04 00:36:45.303994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:45.303957    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5" event={"ID":"13e2500a-38b3-4493-8ce1-84f2c82c01d6","Type":"ContainerStarted","Data":"e644ecb2bd693c07cb7fcd925c6431844a66033c3dd9dbb32b5e8a9e41808534"}
Nov 04 00:36:45.328377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:45.328313    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5" podStartSLOduration=1.328294306 podStartE2EDuration="1.328294306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:36:45.323036272 +0000 UTC m=+4070.422636415" watchObservedRunningTime="2026-11-04 00:36:45.328294306 +0000 UTC m=+4070.427894451"
Nov 04 00:36:45.875421 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:45.874502    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"]
Nov 04 00:36:46.313202 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:46.313163    5074 generic.go:356] "Generic (PLEG): container finished" podID="13e2500a-38b3-4493-8ce1-84f2c82c01d6" containerID="8b14895b44852180aef7688f2816f86a15d3d2a9eb99f99a1468d31a3bc36e68" exitCode=0
Nov 04 00:36:46.313622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:46.313226    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5" event={"ID":"13e2500a-38b3-4493-8ce1-84f2c82c01d6","Type":"ContainerDied","Data":"8b14895b44852180aef7688f2816f86a15d3d2a9eb99f99a1468d31a3bc36e68"}
Nov 04 00:36:47.486492 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.486458    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:47.576110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.576023    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca\") pod \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") "
Nov 04 00:36:47.576309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.576156    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca\") pod \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") "
Nov 04 00:36:47.576309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.576201    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m\" (UniqueName: \"kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m\") pod \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\" (UID: \"13e2500a-38b3-4493-8ce1-84f2c82c01d6\") "
Nov 04 00:36:47.576602 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.576563    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca" pod "13e2500a-38b3-4493-8ce1-84f2c82c01d6" (UID: "13e2500a-38b3-4493-8ce1-84f2c82c01d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:47.576913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.576885    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:47.578953 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.578924    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca" pod "13e2500a-38b3-4493-8ce1-84f2c82c01d6" (UID: "13e2500a-38b3-4493-8ce1-84f2c82c01d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:47.579053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.579022    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m" pod "13e2500a-38b3-4493-8ce1-84f2c82c01d6" (UID: "13e2500a-38b3-4493-8ce1-84f2c82c01d6"). InnerVolumeSpecName "kube-api-access-kx56m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:47.678764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.678480    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e2500a-38b3-4493-8ce1-84f2c82c01d6-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:47.678764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:47.678525    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx56m\" (UniqueName: \"kubernetes.io/projected/13e2500a-38b3-4493-8ce1-84f2c82c01d6-kube-api-access-kx56m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:48.327681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:48.327544    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5" event={"ID":"13e2500a-38b3-4493-8ce1-84f2c82c01d6","Type":"ContainerDied","Data":"e644ecb2bd693c07cb7fcd925c6431844a66033c3dd9dbb32b5e8a9e41808534"}
Nov 04 00:36:48.327681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:48.327583    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"
Nov 04 00:36:48.327681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:48.327607    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e644ecb2bd693c07cb7fcd925c6431844a66033c3dd9dbb32b5e8a9e41808534"
Nov 04 00:36:50.875195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:50.875147    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw" podUID="15bea0eb-57d4-401c-abe4-888841ad40ae" containerName="payload-pre-processing" containerID="cri-o://71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6" gracePeriod=25
Nov 04 00:36:51.048574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.045831    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"]
Nov 04 00:36:51.078830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.078783    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" podUID="07237989-93da-496c-8fe0-60e681a60ded" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:51.204382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.204353    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 04 00:36:51.338323 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.338280    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume\") pod \"15bea0eb-57d4-401c-abe4-888841ad40ae\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") "
Nov 04 00:36:51.338513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.338360    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd\" (UniqueName: \"kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd\") pod \"15bea0eb-57d4-401c-abe4-888841ad40ae\" (UID: \"15bea0eb-57d4-401c-abe4-888841ad40ae\") "
Nov 04 00:36:51.339174 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.339138    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume" pod "15bea0eb-57d4-401c-abe4-888841ad40ae" (UID: "15bea0eb-57d4-401c-abe4-888841ad40ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:51.341250 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.341206    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd" pod "15bea0eb-57d4-401c-abe4-888841ad40ae" (UID: "15bea0eb-57d4-401c-abe4-888841ad40ae"). InnerVolumeSpecName "kube-api-access-ld8cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:51.351629 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.351558    5074 generic.go:356] "Generic (PLEG): container finished" podID="15bea0eb-57d4-401c-abe4-888841ad40ae" containerID="71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6" exitCode=0
Nov 04 00:36:51.351715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.351651    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"
Nov 04 00:36:51.351715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.351678    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw" event={"ID":"15bea0eb-57d4-401c-abe4-888841ad40ae","Type":"ContainerDied","Data":"71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6"}
Nov 04 00:36:51.351853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.351765    5074 scope.go:122] "RemoveContainer" containerID="71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6"
Nov 04 00:36:51.351853 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.351769    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw" event={"ID":"15bea0eb-57d4-401c-abe4-888841ad40ae","Type":"ContainerDied","Data":"a3db5f98558556b1e92dcd76a6e77f5994daea84b0d8876644d52b2881fdc446"}
Nov 04 00:36:51.362616 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.362598    5074 scope.go:122] "RemoveContainer" containerID="71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6"
Nov 04 00:36:51.362977 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:51.362952    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6\": container with ID starting with 71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6 not found: ID does not exist" containerID="71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6"
Nov 04 00:36:51.363078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.362987    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6"} err="failed to get container status \"71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6\": rpc error: code = NotFound desc = could not find container \"71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6\": container with ID starting with 71b6b0048db080edb96b9a2f3d41e49a39b905c20ebccc739a11aa790b1b64c6 not found: ID does not exist"
Nov 04 00:36:51.377429 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.377400    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"]
Nov 04 00:36:51.381591 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.381569    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c60893-564f6c79cf-8cccw"]
Nov 04 00:36:51.440104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.440065    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bea0eb-57d4-401c-abe4-888841ad40ae-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:51.440104 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.440103    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8cd\" (UniqueName: \"kubernetes.io/projected/15bea0eb-57d4-401c-abe4-888841ad40ae-kube-api-access-ld8cd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:51.760656 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:51.760573    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main" containerID="cri-o://ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56" gracePeriod=45
Nov 04 00:36:52.028399 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.028331    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 04 00:36:52.147110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147070    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs\") pod \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") "
Nov 04 00:36:52.147306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147158    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir\") pod \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") "
Nov 04 00:36:52.147306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147186    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm\") pod \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") "
Nov 04 00:36:52.147306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147217    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache\") pod \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") "
Nov 04 00:36:52.147495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147354    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home\") pod \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") "
Nov 04 00:36:52.147495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147399    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2\" (UniqueName: \"kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2\") pod \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\" (UID: \"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3\") "
Nov 04 00:36:52.147495 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147474    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir" pod "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" (UID: "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.147717 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147684    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache" pod "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" (UID: "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.147955 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147935    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.148054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.147957    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.148214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.148191    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home" pod "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" (UID: "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.150238 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.150196    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2" pod "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" (UID: "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3"). InnerVolumeSpecName "kube-api-access-mrgh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:52.150425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.150395    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm" pod "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" (UID: "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.150615 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.150585    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs" pod "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" (UID: "a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:52.249182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.249145    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.249182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.249178    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.249182 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.249188    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrgh2\" (UniqueName: \"kubernetes.io/projected/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-kube-api-access-mrgh2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.249500 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.249200    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.363286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.363190    5074 generic.go:356] "Generic (PLEG): container finished" podID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerID="ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56" exitCode=0
Nov 04 00:36:52.363286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.363251    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"
Nov 04 00:36:52.363541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.363287    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" event={"ID":"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3","Type":"ContainerDied","Data":"ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56"}
Nov 04 00:36:52.363541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.363337    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw" event={"ID":"a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3","Type":"ContainerDied","Data":"52281e04e90bf8b90440191a682636e152f2ab9dd8e9644502c2ca04e1ca82b4"}
Nov 04 00:36:52.363541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.363359    5074 scope.go:122] "RemoveContainer" containerID="ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56"
Nov 04 00:36:52.374382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.374362    5074 scope.go:122] "RemoveContainer" containerID="ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56"
Nov 04 00:36:52.374680 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:52.374659    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56\": container with ID starting with ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56 not found: ID does not exist" containerID="ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56"
Nov 04 00:36:52.374805 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.374689    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56"} err="failed to get container status \"ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56\": rpc error: code = NotFound desc = could not find container \"ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56\": container with ID starting with ad44b5904879aea26fa9414652c126ab202855a38787feca6cdafff11c024b56 not found: ID does not exist"
Nov 04 00:36:52.396208 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.396179    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"]
Nov 04 00:36:52.400135 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:52.400111    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-2-w3-c60893-kserve-5b788b848d-t6nxw"]
Nov 04 00:36:53.017130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:53.017098    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bea0eb-57d4-401c-abe4-888841ad40ae" path="/var/lib/kubelet/pods/15bea0eb-57d4-401c-abe4-888841ad40ae/volumes"
Nov 04 00:36:53.017778 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:53.017729    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" path="/var/lib/kubelet/pods/a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3/volumes"
Nov 04 00:36:56.046812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.046722    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" podUID="4626a9ed-a96d-4640-b657-8334384dc755" containerName="payload-pre-processing" containerID="cri-o://382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6" gracePeriod=25
Nov 04 00:36:56.328904 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.328875    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 04 00:36:56.387362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.387327    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume\") pod \"4626a9ed-a96d-4640-b657-8334384dc755\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") "
Nov 04 00:36:56.387362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.387372    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v\" (UniqueName: \"kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v\") pod \"4626a9ed-a96d-4640-b657-8334384dc755\" (UID: \"4626a9ed-a96d-4640-b657-8334384dc755\") "
Nov 04 00:36:56.388129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.388094    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume" pod "4626a9ed-a96d-4640-b657-8334384dc755" (UID: "4626a9ed-a96d-4640-b657-8334384dc755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:56.390065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.390036    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v" pod "4626a9ed-a96d-4640-b657-8334384dc755" (UID: "4626a9ed-a96d-4640-b657-8334384dc755"). InnerVolumeSpecName "kube-api-access-9lg4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:56.405090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.405064    5074 generic.go:356] "Generic (PLEG): container finished" podID="4626a9ed-a96d-4640-b657-8334384dc755" containerID="382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6" exitCode=0
Nov 04 00:36:56.405212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.405110    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"
Nov 04 00:36:56.405212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.405147    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" event={"ID":"4626a9ed-a96d-4640-b657-8334384dc755","Type":"ContainerDied","Data":"382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6"}
Nov 04 00:36:56.405212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.405201    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" event={"ID":"4626a9ed-a96d-4640-b657-8334384dc755","Type":"ContainerDied","Data":"f5cd4c3e165f5b8bbca5af31c8965c03b58ecb23ae9a197185faff62a1aa9c87"}
Nov 04 00:36:56.405375 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.405228    5074 scope.go:122] "RemoveContainer" containerID="382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6"
Nov 04 00:36:56.417584 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.417563    5074 scope.go:122] "RemoveContainer" containerID="382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6"
Nov 04 00:36:56.417916 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:56.417893    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6\": container with ID starting with 382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6 not found: ID does not exist" containerID="382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6"
Nov 04 00:36:56.418016 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.417922    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6"} err="failed to get container status \"382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6\": rpc error: code = NotFound desc = could not find container \"382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6\": container with ID starting with 382fd4b4adc016b9d52752a87b6ac360989c0bf1dfeea0177a26fc86dbc1a3c6 not found: ID does not exist"
Nov 04 00:36:56.432325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.432298    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"]
Nov 04 00:36:56.439948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.439923    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9"]
Nov 04 00:36:56.488593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.488557    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4626a9ed-a96d-4640-b657-8334384dc755-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:56.488593 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:56.488586    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lg4v\" (UniqueName: \"kubernetes.io/projected/4626a9ed-a96d-4640-b657-8334384dc755-kube-api-access-9lg4v\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:57.016771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.016709    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4626a9ed-a96d-4640-b657-8334384dc755" path="/var/lib/kubelet/pods/4626a9ed-a96d-4640-b657-8334384dc755/volumes"
Nov 04 00:36:57.300827 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.300709    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b3bea7-849bd5c867-t9bm9" podUID="4626a9ed-a96d-4640-b657-8334384dc755" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.180:50052\" within 1s: context deadline exceeded"
Nov 04 00:36:57.300827 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:36:57.300802    5074 logging.go:55] [core] [Channel #25818 SubChannel #25819]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.180:50052", ServerName: "10.132.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.180:50052: operation was canceled"
Nov 04 00:36:57.728156 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.728126    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:36:57.800098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.800058    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6\" (UniqueName: \"kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6\") pod \"07237989-93da-496c-8fe0-60e681a60ded\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") "
Nov 04 00:36:57.802258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.802230    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6" pod "07237989-93da-496c-8fe0-60e681a60ded" (UID: "07237989-93da-496c-8fe0-60e681a60ded"). InnerVolumeSpecName "kube-api-access-42hc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:57.900976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.900938    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume\") pod \"07237989-93da-496c-8fe0-60e681a60ded\" (UID: \"07237989-93da-496c-8fe0-60e681a60ded\") "
Nov 04 00:36:57.901251 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.901236    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hc6\" (UniqueName: \"kubernetes.io/projected/07237989-93da-496c-8fe0-60e681a60ded-kube-api-access-42hc6\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:57.901371 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:57.901343    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume" pod "07237989-93da-496c-8fe0-60e681a60ded" (UID: "07237989-93da-496c-8fe0-60e681a60ded"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:58.002004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.001969    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07237989-93da-496c-8fe0-60e681a60ded-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:58.023379 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:58.023340    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:36:58.422157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.422122    5074 generic.go:356] "Generic (PLEG): container finished" podID="07237989-93da-496c-8fe0-60e681a60ded" containerID="463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd" exitCode=0
Nov 04 00:36:58.422702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.422173    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"
Nov 04 00:36:58.422702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.422207    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" event={"ID":"07237989-93da-496c-8fe0-60e681a60ded","Type":"ContainerDied","Data":"463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd"}
Nov 04 00:36:58.422702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.422267    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf" event={"ID":"07237989-93da-496c-8fe0-60e681a60ded","Type":"ContainerDied","Data":"bbc75ea9108afcec5fd1c99e3739ef0b00a1acadf003260c4ea4231c438dc5ff"}
Nov 04 00:36:58.422702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.422293    5074 scope.go:122] "RemoveContainer" containerID="463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd"
Nov 04 00:36:58.433667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.433649    5074 scope.go:122] "RemoveContainer" containerID="463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd"
Nov 04 00:36:58.433969 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:36:58.433947    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd\": container with ID starting with 463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd not found: ID does not exist" containerID="463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd"
Nov 04 00:36:58.434038 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.433976    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd"} err="failed to get container status \"463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd\": rpc error: code = NotFound desc = could not find container \"463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd\": container with ID starting with 463fff56cb0bee96a4398133567509a18844aa35b932068c1f246b045e6f16fd not found: ID does not exist"
Nov 04 00:36:58.449405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.449380    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"]
Nov 04 00:36:58.452566 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.452543    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ade4-5455c85595-bbclf"]
Nov 04 00:36:58.855386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.855349    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"]
Nov 04 00:36:58.856053 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856028    5074 cpu_manager.go:414] "containerMap: removing container" podUID="15bea0eb-57d4-401c-abe4-888841ad40ae" containerName="payload-pre-processing"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856055    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bea0eb-57d4-401c-abe4-888841ad40ae" containerName="payload-pre-processing"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856076    5074 cpu_manager.go:414] "containerMap: removing container" podUID="07237989-93da-496c-8fe0-60e681a60ded" containerName="payload-pre-processing"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856085    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07237989-93da-496c-8fe0-60e681a60ded" containerName="payload-pre-processing"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856105    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4626a9ed-a96d-4640-b657-8334384dc755" containerName="payload-pre-processing"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856112    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4626a9ed-a96d-4640-b657-8334384dc755" containerName="payload-pre-processing"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856126    5074 cpu_manager.go:414] "containerMap: removing container" podUID="13e2500a-38b3-4493-8ce1-84f2c82c01d6" containerName="revoke-keys"
Nov 04 00:36:58.856173 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856134    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e2500a-38b3-4493-8ce1-84f2c82c01d6" containerName="revoke-keys"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856182    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856190    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856293    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a65ffa-55ad-47f5-a293-9d2ac3ae27a3" containerName="main"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856310    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bea0eb-57d4-401c-abe4-888841ad40ae" containerName="payload-pre-processing"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856321    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="07237989-93da-496c-8fe0-60e681a60ded" containerName="payload-pre-processing"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856332    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4626a9ed-a96d-4640-b657-8334384dc755" containerName="payload-pre-processing"
Nov 04 00:36:58.856571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.856342    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e2500a-38b3-4493-8ce1-84f2c82c01d6" containerName="revoke-keys"
Nov 04 00:36:58.861444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.861419    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.874423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.874390    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"]
Nov 04 00:36:58.909396 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.909357    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.909567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.909423    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.909567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.909515    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.909567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.909556    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.909992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.909689    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnps4\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.919027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.910373    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.919027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.910455    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.919027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.910512    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.919027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.910559    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:58.950702 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:58.950666    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\""
Nov 04 00:36:59.011236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011203    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011248    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011295    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011346    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011388 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011372    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011425    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnps4\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011459    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011509    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011577    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011677    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.011954 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011914    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.012011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.011977    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.012195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.012172    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.012572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.012551    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.014410 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.014385    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.014521 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.014441    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.019273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.019248    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07237989-93da-496c-8fe0-60e681a60ded" path="/var/lib/kubelet/pods/07237989-93da-496c-8fe0-60e681a60ded/volumes"
Nov 04 00:36:59.024466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.024440    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.024921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.024896    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnps4\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.180999 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.180901    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:36:59.468283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.468249    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"]
Nov 04 00:36:59.469421 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:36:59.469389    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc11594e_dbe3_4cc8_9270_383f319fe5f5.slice/crio-798c2e778612101a17aedf9f68f1fd3f9e71f6d7635817446b854baecb508c2d WatchSource:0}: Error finding container 798c2e778612101a17aedf9f68f1fd3f9e71f6d7635817446b854baecb508c2d: Status 404 returned error can't find the container with id 798c2e778612101a17aedf9f68f1fd3f9e71f6d7635817446b854baecb508c2d
Nov 04 00:36:59.471812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.471780    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:36:59.471921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.471865    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:36:59.471921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:36:59.471909    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:37:00.439282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:00.439241    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" event={"ID":"cc11594e-dbe3-4cc8-9270-383f319fe5f5","Type":"ContainerStarted","Data":"0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea"}
Nov 04 00:37:00.439506 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:00.439310    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" event={"ID":"cc11594e-dbe3-4cc8-9270-383f319fe5f5","Type":"ContainerStarted","Data":"798c2e778612101a17aedf9f68f1fd3f9e71f6d7635817446b854baecb508c2d"}
Nov 04 00:37:00.469408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:00.469358    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" podStartSLOduration=2.469345741 podStartE2EDuration="2.469345741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:37:00.46469646 +0000 UTC m=+4085.564296639" watchObservedRunningTime="2026-11-04 00:37:00.469345741 +0000 UTC m=+4085.568945885"
Nov 04 00:37:01.181926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:01.181889    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:37:01.187841 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:01.187814    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:37:01.449027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:01.447676    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:37:01.449027 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:01.448889    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:37:02.011776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:02.011647    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"]
Nov 04 00:37:02.453470 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:02.453434    5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" secret="" err="secret \"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\" not found"
Nov 04 00:37:03.460261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:03.460224    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" podUID="cc11594e-dbe3-4cc8-9270-383f319fe5f5" containerName="istio-proxy" containerID="cri-o://0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea" gracePeriod=30
Nov 04 00:37:08.737387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.737316    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:37:08.833254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833208    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833267    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833367    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833497    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833548    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833582    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833641    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833708    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.833802 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.833735    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy\") pod \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\" (UID: \"cc11594e-dbe3-4cc8-9270-383f319fe5f5\") "
Nov 04 00:37:08.834321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.834007    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.834321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.834185    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.834321 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.834252    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:08.834595 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.834379    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.834757 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.834716    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.835009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.834974    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.836361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.836331    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:08.837257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.837212    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:37:08.837379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.837287    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.838337 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.838304    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4" pod "cc11594e-dbe3-4cc8-9270-383f319fe5f5" (UID: "cc11594e-dbe3-4cc8-9270-383f319fe5f5"). InnerVolumeSpecName "kube-api-access-fnps4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:08.935279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935235    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935274    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnps4\" (UniqueName: \"kubernetes.io/projected/cc11594e-dbe3-4cc8-9270-383f319fe5f5-kube-api-access-fnps4\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935279 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935285    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935293    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935303    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935311    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc11594e-dbe3-4cc8-9270-383f319fe5f5-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935321    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.935554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:08.935329    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc11594e-dbe3-4cc8-9270-383f319fe5f5-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:09.500035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.499956    5074 generic.go:356] "Generic (PLEG): container finished" podID="cc11594e-dbe3-4cc8-9270-383f319fe5f5" containerID="0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea" exitCode=0
Nov 04 00:37:09.500035 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.500027    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"
Nov 04 00:37:09.500325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.500157    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" event={"ID":"cc11594e-dbe3-4cc8-9270-383f319fe5f5","Type":"ContainerDied","Data":"0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea"}
Nov 04 00:37:09.500325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.500226    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9" event={"ID":"cc11594e-dbe3-4cc8-9270-383f319fe5f5","Type":"ContainerDied","Data":"798c2e778612101a17aedf9f68f1fd3f9e71f6d7635817446b854baecb508c2d"}
Nov 04 00:37:09.500325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.500256    5074 scope.go:122] "RemoveContainer" containerID="0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea"
Nov 04 00:37:09.511025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.511005    5074 scope.go:122] "RemoveContainer" containerID="0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea"
Nov 04 00:37:09.511357 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:37:09.511331    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea\": container with ID starting with 0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea not found: ID does not exist" containerID="0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea"
Nov 04 00:37:09.511444 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.511364    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea"} err="failed to get container status \"0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea\": rpc error: code = NotFound desc = could not find container \"0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea\": container with ID starting with 0e7c9b834f6413b365c5875537e4411bc2299e7ad4fc2cbc043d060537bb75ea not found: ID does not exist"
Nov 04 00:37:09.529157 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.529126    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"]
Nov 04 00:37:09.533635 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:09.533611    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-lb4p9"]
Nov 04 00:37:10.117475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:10.117436    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"]
Nov 04 00:37:10.126535 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:10.126502    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"]
Nov 04 00:37:11.017366 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:11.017325    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc11594e-dbe3-4cc8-9270-383f319fe5f5" path="/var/lib/kubelet/pods/cc11594e-dbe3-4cc8-9270-383f319fe5f5/volumes"
Nov 04 00:37:13.074638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.074593    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"]
Nov 04 00:37:13.075306 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.075283    5074 cpu_manager.go:414] "containerMap: removing container" podUID="cc11594e-dbe3-4cc8-9270-383f319fe5f5" containerName="istio-proxy"
Nov 04 00:37:13.075379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.075309    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc11594e-dbe3-4cc8-9270-383f319fe5f5" containerName="istio-proxy"
Nov 04 00:37:13.075488 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.075471    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc11594e-dbe3-4cc8-9270-383f319fe5f5" containerName="istio-proxy"
Nov 04 00:37:13.081129 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.081103    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.097386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.096690    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"]
Nov 04 00:37:13.167288 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.167249    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c60893-openshift-default-dockercfg-hwjgp\""
Nov 04 00:37:13.182782 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.182723    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.182860 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.182805    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.182900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.182857    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.182986 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.182962    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.183033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.183009    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.183117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.183102    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.183170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.183134    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhsj\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.183256 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.183234    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.183309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.183281    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.283942 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.283904    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhsj\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.283963    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.283988    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284043    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284064    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284168 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284100    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284173    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284212    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284243    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284395    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284480 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284451    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.284918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.284889    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.285084 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.285060    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.285167 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.285139    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.286573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.286547    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.287478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.287459    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.298001 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.297973    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.298130 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.298096    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhsj\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.400075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.399972    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:13.695096 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:37:13.695065    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9806d1cc_7243_4631_9805_1792e5a9ce2d.slice/crio-bfa5c598b66ed684699dab5629523da1437f326adbac7cacbc6c3448fb94e168 WatchSource:0}: Error finding container bfa5c598b66ed684699dab5629523da1437f326adbac7cacbc6c3448fb94e168: Status 404 returned error can't find the container with id bfa5c598b66ed684699dab5629523da1437f326adbac7cacbc6c3448fb94e168
Nov 04 00:37:13.696397 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.696369    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"]
Nov 04 00:37:13.697585 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.697552    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:37:13.697693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.697626    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:37:13.697693 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:13.697670    5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 04 00:37:14.540201 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:14.540156    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" event={"ID":"9806d1cc-7243-4631-9805-1792e5a9ce2d","Type":"ContainerStarted","Data":"17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af"}
Nov 04 00:37:14.540649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:14.540223    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" event={"ID":"9806d1cc-7243-4631-9805-1792e5a9ce2d","Type":"ContainerStarted","Data":"bfa5c598b66ed684699dab5629523da1437f326adbac7cacbc6c3448fb94e168"}
Nov 04 00:37:14.568379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:14.568328    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" podStartSLOduration=1.568314808 podStartE2EDuration="1.568314808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:37:14.56614882 +0000 UTC m=+4099.665748968" watchObservedRunningTime="2026-11-04 00:37:14.568314808 +0000 UTC m=+4099.667914952"
Nov 04 00:37:15.127763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.122810    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw" podUID="ea32b661-697a-48fb-8bc2-b950be89f0d4" containerName="payload-processing" containerID="cri-o://c9e36cd2d992e9ef97c149317f283fed43e3edf3865d55f90e2cc6654979b43c" gracePeriod=25
Nov 04 00:37:15.128914 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.128836    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r" podUID="4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" containerName="payload-pre-processing" containerID="cri-o://9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775" gracePeriod=25
Nov 04 00:37:15.431476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.431380    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:15.442301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.441968    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body=
Nov 04 00:37:15.442301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.442036    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused"
Nov 04 00:37:15.490539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.490512    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 04 00:37:15.571119 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.571084    5074 generic.go:356] "Generic (PLEG): container finished" podID="ea32b661-697a-48fb-8bc2-b950be89f0d4" containerID="c9e36cd2d992e9ef97c149317f283fed43e3edf3865d55f90e2cc6654979b43c" exitCode=0
Nov 04 00:37:15.571482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.571267    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw" event={"ID":"ea32b661-697a-48fb-8bc2-b950be89f0d4","Type":"ContainerDied","Data":"c9e36cd2d992e9ef97c149317f283fed43e3edf3865d55f90e2cc6654979b43c"}
Nov 04 00:37:15.571482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.571305    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw" event={"ID":"ea32b661-697a-48fb-8bc2-b950be89f0d4","Type":"ContainerDied","Data":"b05cfb7aa7291ddc57d53c054dabfb2eb11158d546f409c222e9428d789a207e"}
Nov 04 00:37:15.571482 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.571327    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b05cfb7aa7291ddc57d53c054dabfb2eb11158d546f409c222e9428d789a207e"
Nov 04 00:37:15.574609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.574585    5074 generic.go:356] "Generic (PLEG): container finished" podID="4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" containerID="9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775" exitCode=0
Nov 04 00:37:15.575350 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.574871    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"
Nov 04 00:37:15.575535 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.574897    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r" event={"ID":"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff","Type":"ContainerDied","Data":"9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775"}
Nov 04 00:37:15.575603 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.575569    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r" event={"ID":"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff","Type":"ContainerDied","Data":"a2155ed34972fce624b900e53366691d6560ff461a624f304cac3f5887950da8"}
Nov 04 00:37:15.575646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.575600    5074 scope.go:122] "RemoveContainer" containerID="9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775"
Nov 04 00:37:15.576487 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.576443    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 04 00:37:15.590678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.590660    5074 scope.go:122] "RemoveContainer" containerID="9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775"
Nov 04 00:37:15.591120 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:37:15.591092    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775\": container with ID starting with 9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775 not found: ID does not exist" containerID="9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775"
Nov 04 00:37:15.591194 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.591131    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775"} err="failed to get container status \"9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775\": rpc error: code = NotFound desc = could not find container \"9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775\": container with ID starting with 9ba2ae6e69658dfe98972a445451b6f003901054232d723429da3c3254ce1775 not found: ID does not exist"
Nov 04 00:37:15.639159 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.639117    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume\") pod \"ea32b661-697a-48fb-8bc2-b950be89f0d4\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") "
Nov 04 00:37:15.639361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.639245    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume\") pod \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") "
Nov 04 00:37:15.639361 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.639286    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw\" (UniqueName: \"kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw\") pod \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\" (UID: \"4efd4e7c-27b4-4119-b2eb-67d7a512f8ff\") "
Nov 04 00:37:15.639489 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.639356    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c\" (UniqueName: \"kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c\") pod \"ea32b661-697a-48fb-8bc2-b950be89f0d4\" (UID: \"ea32b661-697a-48fb-8bc2-b950be89f0d4\") "
Nov 04 00:37:15.640601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.640567    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume" pod "ea32b661-697a-48fb-8bc2-b950be89f0d4" (UID: "ea32b661-697a-48fb-8bc2-b950be89f0d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:15.640848 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.640797    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume" pod "4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" (UID: "4efd4e7c-27b4-4119-b2eb-67d7a512f8ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:15.641428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.641229    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea32b661-697a-48fb-8bc2-b950be89f0d4-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:15.641428 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.641255    5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-plugins-config-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:15.643196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.642855    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c" pod "ea32b661-697a-48fb-8bc2-b950be89f0d4" (UID: "ea32b661-697a-48fb-8bc2-b950be89f0d4"). InnerVolumeSpecName "kube-api-access-7hs9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:15.647075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.646203    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw" pod "4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" (UID: "4efd4e7c-27b4-4119-b2eb-67d7a512f8ff"). InnerVolumeSpecName "kube-api-access-lj6lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:15.742088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.742037    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj6lw\" (UniqueName: \"kubernetes.io/projected/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff-kube-api-access-lj6lw\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:15.742088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.742084    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hs9c\" (UniqueName: \"kubernetes.io/projected/ea32b661-697a-48fb-8bc2-b950be89f0d4-kube-api-access-7hs9c\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:15.932417 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.932322    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"]
Nov 04 00:37:15.938212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:15.938171    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-15b02a-749545846c-49m5r"]
Nov 04 00:37:16.403207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:16.403160    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body=
Nov 04 00:37:16.403529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:16.403495    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused"
Nov 04 00:37:16.585667 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:16.585625    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"
Nov 04 00:37:16.627896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:16.627800    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"]
Nov 04 00:37:16.640143 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:16.640100    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-15b02a-7c884957fb-zhztw"]
Nov 04 00:37:16.853485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:16.853423    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"]
Nov 04 00:37:17.036483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:17.036437    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" path="/var/lib/kubelet/pods/4efd4e7c-27b4-4119-b2eb-67d7a512f8ff/volumes"
Nov 04 00:37:17.037079 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:17.037055    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea32b661-697a-48fb-8bc2-b950be89f0d4" path="/var/lib/kubelet/pods/ea32b661-697a-48fb-8bc2-b950be89f0d4/volumes"
Nov 04 00:37:17.406257 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:17.406209    5074 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body=
Nov 04 00:37:17.407050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:17.406589    5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused"
Nov 04 00:37:17.591892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:17.591851    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy" containerID="cri-o://17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af" gracePeriod=30
Nov 04 00:37:20.261463 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.261429    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"]
Nov 04 00:37:20.261917 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.261903    5074 cpu_manager.go:414] "containerMap: removing container" podUID="ea32b661-697a-48fb-8bc2-b950be89f0d4" containerName="payload-processing"
Nov 04 00:37:20.261965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.261918    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea32b661-697a-48fb-8bc2-b950be89f0d4" containerName="payload-processing"
Nov 04 00:37:20.261965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.261930    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" containerName="payload-pre-processing"
Nov 04 00:37:20.261965 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.261937    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" containerName="payload-pre-processing"
Nov 04 00:37:20.262074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.262002    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea32b661-697a-48fb-8bc2-b950be89f0d4" containerName="payload-processing"
Nov 04 00:37:20.262074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.262010    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efd4e7c-27b4-4119-b2eb-67d7a512f8ff" containerName="payload-pre-processing"
Nov 04 00:37:20.266770 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.266748    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.271688 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.271661    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:37:20.271832 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.271809    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:37:20.274147 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.274122    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"]
Nov 04 00:37:20.394990 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.394954    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.395170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.395088    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.395170 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.395158    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjv7\" (UniqueName: \"kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.496091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.496052    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjv7\" (UniqueName: \"kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.496280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.496116    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.496280 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.496197    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.496720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.496699    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.498818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.498792    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.517840 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.517778    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjv7\" (UniqueName: \"kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7\") pod \"maas-api-revoke-sub-787c5223-mkqdp\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.583532 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.583498    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:20.871177 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:20.871146    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"]
Nov 04 00:37:20.871859 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:37:20.871830    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb982ee71_bf94_4dbf_a23a_c845080d96eb.slice/crio-c47e181663e9e5a6e82bfc7ee2d578fe6a902e5ea4c8a0c4be6314dc22f8d6bf WatchSource:0}: Error finding container c47e181663e9e5a6e82bfc7ee2d578fe6a902e5ea4c8a0c4be6314dc22f8d6bf: Status 404 returned error can't find the container with id c47e181663e9e5a6e82bfc7ee2d578fe6a902e5ea4c8a0c4be6314dc22f8d6bf
Nov 04 00:37:21.625078 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:21.625033    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp" event={"ID":"b982ee71-bf94-4dbf-a23a-c845080d96eb","Type":"ContainerStarted","Data":"eda2f3bf2d82d0bf8a2dcca1a0e35296cce7d258e542c372146b080829e35556"}
Nov 04 00:37:21.625559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:21.625108    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp" event={"ID":"b982ee71-bf94-4dbf-a23a-c845080d96eb","Type":"ContainerStarted","Data":"c47e181663e9e5a6e82bfc7ee2d578fe6a902e5ea4c8a0c4be6314dc22f8d6bf"}
Nov 04 00:37:21.645152 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:21.645103    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp" podStartSLOduration=1.645089064 podStartE2EDuration="1.645089064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:37:21.644557197 +0000 UTC m=+4106.744157342" watchObservedRunningTime="2026-11-04 00:37:21.645089064 +0000 UTC m=+4106.744689207"
Nov 04 00:37:22.633440 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:22.633400    5074 generic.go:356] "Generic (PLEG): container finished" podID="b982ee71-bf94-4dbf-a23a-c845080d96eb" containerID="eda2f3bf2d82d0bf8a2dcca1a0e35296cce7d258e542c372146b080829e35556" exitCode=0
Nov 04 00:37:22.633920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:22.633475    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp" event={"ID":"b982ee71-bf94-4dbf-a23a-c845080d96eb","Type":"ContainerDied","Data":"eda2f3bf2d82d0bf8a2dcca1a0e35296cce7d258e542c372146b080829e35556"}
Nov 04 00:37:23.043730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.043694    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"]
Nov 04 00:37:23.046519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.046493    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fac65e7-9z6kb"]
Nov 04 00:37:23.807364 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.807334    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:23.936570 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.935027    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca\") pod \"b982ee71-bf94-4dbf-a23a-c845080d96eb\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") "
Nov 04 00:37:23.936570 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.935167    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7\" (UniqueName: \"kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7\") pod \"b982ee71-bf94-4dbf-a23a-c845080d96eb\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") "
Nov 04 00:37:23.936570 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.935260    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca\") pod \"b982ee71-bf94-4dbf-a23a-c845080d96eb\" (UID: \"b982ee71-bf94-4dbf-a23a-c845080d96eb\") "
Nov 04 00:37:23.937812 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.937763    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca" pod "b982ee71-bf94-4dbf-a23a-c845080d96eb" (UID: "b982ee71-bf94-4dbf-a23a-c845080d96eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:23.941844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.941790    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca" pod "b982ee71-bf94-4dbf-a23a-c845080d96eb" (UID: "b982ee71-bf94-4dbf-a23a-c845080d96eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:23.950934 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:23.950897    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7" pod "b982ee71-bf94-4dbf-a23a-c845080d96eb" (UID: "b982ee71-bf94-4dbf-a23a-c845080d96eb"). InnerVolumeSpecName "kube-api-access-dsjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:24.036854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.036819    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjv7\" (UniqueName: \"kubernetes.io/projected/b982ee71-bf94-4dbf-a23a-c845080d96eb-kube-api-access-dsjv7\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.036854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.036849    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.036854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.036859    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ee71-bf94-4dbf-a23a-c845080d96eb-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.649909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.649864    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp" event={"ID":"b982ee71-bf94-4dbf-a23a-c845080d96eb","Type":"ContainerDied","Data":"c47e181663e9e5a6e82bfc7ee2d578fe6a902e5ea4c8a0c4be6314dc22f8d6bf"}
Nov 04 00:37:24.649909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.649902    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"
Nov 04 00:37:24.649909 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.649915    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c47e181663e9e5a6e82bfc7ee2d578fe6a902e5ea4c8a0c4be6314dc22f8d6bf"
Nov 04 00:37:24.747750 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:24.747704    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"]
Nov 04 00:37:25.017730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.017694    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fe2897-75b6-4c46-a5b6-db49e6969dc4" path="/var/lib/kubelet/pods/c6fe2897-75b6-4c46-a5b6-db49e6969dc4/volumes"
Nov 04 00:37:25.369123 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.369097    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:25.451980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.451939    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452001    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452073    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452169    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452182    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.452355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452206    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452239    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452268    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452317    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452362    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket\") pod \"9806d1cc-7243-4631-9805-1792e5a9ce2d\" (UID: \"9806d1cc-7243-4631-9805-1792e5a9ce2d\") "
Nov 04 00:37:25.452622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452480    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.452622 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452489    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:25.452829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452797    5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-data\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.452829 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452817    5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-credential-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.452945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452821    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.452945 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452831    5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9806d1cc-7243-4631-9805-1792e5a9ce2d-istiod-ca-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.453046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.452988    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.454665 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.454637    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "kube-api-access-zxhsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:25.455024 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.454993    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.455158 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.455137    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:37:25.455307 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.455283    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token" pod "9806d1cc-7243-4631-9805-1792e5a9ce2d" (UID: "9806d1cc-7243-4631-9805-1792e5a9ce2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:25.553466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.553386    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxhsj\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-kube-api-access-zxhsj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.553466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.553415    5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.553466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.553426    5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-podinfo\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.553466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.553435    5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-token\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.553466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.553443    5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-istio-envoy\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.553466 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.553453    5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9806d1cc-7243-4631-9805-1792e5a9ce2d-workload-socket\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.657580 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.657543    5074 generic.go:356] "Generic (PLEG): container finished" podID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerID="17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af" exitCode=0
Nov 04 00:37:25.657807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.657603    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"
Nov 04 00:37:25.657807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.657624    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" event={"ID":"9806d1cc-7243-4631-9805-1792e5a9ce2d","Type":"ContainerDied","Data":"17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af"}
Nov 04 00:37:25.657807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.657682    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt" event={"ID":"9806d1cc-7243-4631-9805-1792e5a9ce2d","Type":"ContainerDied","Data":"bfa5c598b66ed684699dab5629523da1437f326adbac7cacbc6c3448fb94e168"}
Nov 04 00:37:25.657807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.657703    5074 scope.go:122] "RemoveContainer" containerID="17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af"
Nov 04 00:37:25.669161 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.669141    5074 scope.go:122] "RemoveContainer" containerID="17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af"
Nov 04 00:37:25.669465 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:37:25.669439    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af\": container with ID starting with 17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af not found: ID does not exist" containerID="17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af"
Nov 04 00:37:25.669538 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.669478    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af"} err="failed to get container status \"17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af\": rpc error: code = NotFound desc = could not find container \"17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af\": container with ID starting with 17d5d3f808a38ecd7403f2250660f9c140d3c4e41329498d7e722713cd4d08af not found: ID does not exist"
Nov 04 00:37:25.685332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.685299    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"]
Nov 04 00:37:25.695005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:25.690186    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-w7mgt"]
Nov 04 00:37:27.019630 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:27.019424    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" path="/var/lib/kubelet/pods/9806d1cc-7243-4631-9805-1792e5a9ce2d/volumes"
Nov 04 00:37:28.020754 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:37:28.020702    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:37:32.906320 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906285    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"]
Nov 04 00:37:32.908583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906792    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b982ee71-bf94-4dbf-a23a-c845080d96eb" containerName="revoke-keys"
Nov 04 00:37:32.908583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906808    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b982ee71-bf94-4dbf-a23a-c845080d96eb" containerName="revoke-keys"
Nov 04 00:37:32.908583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906839    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy"
Nov 04 00:37:32.908583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906845    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy"
Nov 04 00:37:32.908583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906908    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b982ee71-bf94-4dbf-a23a-c845080d96eb" containerName="revoke-keys"
Nov 04 00:37:32.908583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.906923    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9806d1cc-7243-4631-9805-1792e5a9ce2d" containerName="istio-proxy"
Nov 04 00:37:32.910139 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.910118    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:32.916302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.916273    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:37:32.916529 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.916486    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:37:32.924906 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:32.924874    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"]
Nov 04 00:37:33.030923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.030882    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8ql\" (UniqueName: \"kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.031196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.031160    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.031277 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.031263    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.132779 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.132718    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.132988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.132859    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.132988 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.132935    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8ql\" (UniqueName: \"kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.133943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.133916    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.135606 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.135579    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.147297 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.147265    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8ql\" (UniqueName: \"kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql\") pod \"maas-api-revoke-sub-06ddb2ad-4cqn5\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.230327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.230287    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:33.515458 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.515425    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"]
Nov 04 00:37:33.518055 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:37:33.518012    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ac2b6e_1c6d_4abe_894f_2f45ac1492b2.slice/crio-2408c279959cd1f1a713ea8139af9566c6f71ae9d00cb7eca9dff35ad12dcfda WatchSource:0}: Error finding container 2408c279959cd1f1a713ea8139af9566c6f71ae9d00cb7eca9dff35ad12dcfda: Status 404 returned error can't find the container with id 2408c279959cd1f1a713ea8139af9566c6f71ae9d00cb7eca9dff35ad12dcfda
Nov 04 00:37:33.719262 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:33.719221    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5" event={"ID":"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2","Type":"ContainerStarted","Data":"2408c279959cd1f1a713ea8139af9566c6f71ae9d00cb7eca9dff35ad12dcfda"}
Nov 04 00:37:34.028243 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:34.028207    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"]
Nov 04 00:37:34.035115 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:34.035083    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfa9eba-hsb7p"]
Nov 04 00:37:34.727970 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:34.727932    5074 generic.go:356] "Generic (PLEG): container finished" podID="36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" containerID="8486396b39f081125e46c2012eb06f8c64d14cb70bbf337fe24b8e5b3d93d42b" exitCode=0
Nov 04 00:37:34.728131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:34.727982    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5" event={"ID":"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2","Type":"ContainerDied","Data":"8486396b39f081125e46c2012eb06f8c64d14cb70bbf337fe24b8e5b3d93d42b"}
Nov 04 00:37:35.017400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:35.017365    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5385bcd5-a251-4445-9d38-b99d94c1b17e" path="/var/lib/kubelet/pods/5385bcd5-a251-4445-9d38-b99d94c1b17e/volumes"
Nov 04 00:37:35.892011 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:35.891985    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:36.061834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.061728    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql\" (UniqueName: \"kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql\") pod \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") "
Nov 04 00:37:36.061834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.061801    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca\") pod \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") "
Nov 04 00:37:36.062083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.061901    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca\") pod \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\" (UID: \"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2\") "
Nov 04 00:37:36.062435 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.062402    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca" pod "36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" (UID: "36ac2b6e-1c6d-4abe-894f-2f45ac1492b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:36.063913 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.063885    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca" pod "36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" (UID: "36ac2b6e-1c6d-4abe-894f-2f45ac1492b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:36.064030 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.063975    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql" pod "36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" (UID: "36ac2b6e-1c6d-4abe-894f-2f45ac1492b2"). InnerVolumeSpecName "kube-api-access-pw8ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:36.163522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.163488    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8ql\" (UniqueName: \"kubernetes.io/projected/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-kube-api-access-pw8ql\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:36.163522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.163521    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:36.163733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.163533    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:36.741872 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.741836    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"
Nov 04 00:37:36.742076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.741871    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5" event={"ID":"36ac2b6e-1c6d-4abe-894f-2f45ac1492b2","Type":"ContainerDied","Data":"2408c279959cd1f1a713ea8139af9566c6f71ae9d00cb7eca9dff35ad12dcfda"}
Nov 04 00:37:36.742076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:36.741921    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2408c279959cd1f1a713ea8139af9566c6f71ae9d00cb7eca9dff35ad12dcfda"
Nov 04 00:37:39.768478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:39.768413    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main" containerID="cri-o://57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a" gracePeriod=45
Nov 04 00:37:40.025405 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.025343    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 04 00:37:40.118328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118294    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs\") pod \"c631f8d7-c832-4e70-b291-4e8629e024a5\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") "
Nov 04 00:37:40.118536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118384    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache\") pod \"c631f8d7-c832-4e70-b291-4e8629e024a5\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") "
Nov 04 00:37:40.118536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118423    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv\" (UniqueName: \"kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv\") pod \"c631f8d7-c832-4e70-b291-4e8629e024a5\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") "
Nov 04 00:37:40.118536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118442    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir\") pod \"c631f8d7-c832-4e70-b291-4e8629e024a5\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") "
Nov 04 00:37:40.118536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118467    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm\") pod \"c631f8d7-c832-4e70-b291-4e8629e024a5\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") "
Nov 04 00:37:40.118536 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118527    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home\") pod \"c631f8d7-c832-4e70-b291-4e8629e024a5\" (UID: \"c631f8d7-c832-4e70-b291-4e8629e024a5\") "
Nov 04 00:37:40.118896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118712    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache" pod "c631f8d7-c832-4e70-b291-4e8629e024a5" (UID: "c631f8d7-c832-4e70-b291-4e8629e024a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.118960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118912    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir" pod "c631f8d7-c832-4e70-b291-4e8629e024a5" (UID: "c631f8d7-c832-4e70-b291-4e8629e024a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.119023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.118973    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.119099 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.119078    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home" pod "c631f8d7-c832-4e70-b291-4e8629e024a5" (UID: "c631f8d7-c832-4e70-b291-4e8629e024a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.121121 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.121090    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv" pod "c631f8d7-c832-4e70-b291-4e8629e024a5" (UID: "c631f8d7-c832-4e70-b291-4e8629e024a5"). InnerVolumeSpecName "kube-api-access-zhcqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:40.121353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.121326    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs" pod "c631f8d7-c832-4e70-b291-4e8629e024a5" (UID: "c631f8d7-c832-4e70-b291-4e8629e024a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:40.121539 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.121517    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm" pod "c631f8d7-c832-4e70-b291-4e8629e024a5" (UID: "c631f8d7-c832-4e70-b291-4e8629e024a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.219721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.219677    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c631f8d7-c832-4e70-b291-4e8629e024a5-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.219721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.219717    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcqv\" (UniqueName: \"kubernetes.io/projected/c631f8d7-c832-4e70-b291-4e8629e024a5-kube-api-access-zhcqv\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.219721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.219732    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.219994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.219772    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.219994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.219783    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c631f8d7-c832-4e70-b291-4e8629e024a5-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.768810 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.768774    5074 generic.go:356] "Generic (PLEG): container finished" podID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerID="57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a" exitCode=0
Nov 04 00:37:40.769273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.768828    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"
Nov 04 00:37:40.769273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.768851    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" event={"ID":"c631f8d7-c832-4e70-b291-4e8629e024a5","Type":"ContainerDied","Data":"57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a"}
Nov 04 00:37:40.769273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.768891    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w" event={"ID":"c631f8d7-c832-4e70-b291-4e8629e024a5","Type":"ContainerDied","Data":"0dae702945e4d67df117d141c91d3c75d5e40712971151150843a90493301991"}
Nov 04 00:37:40.769273 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.768908    5074 scope.go:122] "RemoveContainer" containerID="57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a"
Nov 04 00:37:40.778989 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.778975    5074 scope.go:122] "RemoveContainer" containerID="57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a"
Nov 04 00:37:40.779251 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:37:40.779226    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a\": container with ID starting with 57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a not found: ID does not exist" containerID="57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a"
Nov 04 00:37:40.779355 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.779256    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a"} err="failed to get container status \"57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a\": rpc error: code = NotFound desc = could not find container \"57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a\": container with ID starting with 57d683e56fb01b38455ea92f5dbde73bfc3ed2b514947270fe129c8a59aa033a not found: ID does not exist"
Nov 04 00:37:40.794058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.794035    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"]
Nov 04 00:37:40.798113 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:40.798090    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-w4-80f8ba-kserve-5b49b6f7c4-cnk2w"]
Nov 04 00:37:41.020228 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:41.020151    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" path="/var/lib/kubelet/pods/c631f8d7-c832-4e70-b291-4e8629e024a5/volumes"
Nov 04 00:37:55.028185 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:55.028154    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"]
Nov 04 00:37:55.030751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:55.030721    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08db5e8c-nvh2h"]
Nov 04 00:37:57.015642 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:37:57.015606    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5df399a-0978-4e85-9de6-3e4cd6c21c9d" path="/var/lib/kubelet/pods/e5df399a-0978-4e85-9de6-3e4cd6c21c9d/volumes"
Nov 04 00:37:58.020695 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:37:58.020657    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:38:01.805340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:01.805304    5074 scope.go:122] "RemoveContainer" containerID="c9e36cd2d992e9ef97c149317f283fed43e3edf3865d55f90e2cc6654979b43c"
Nov 04 00:38:04.245245 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:04.245213    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qnvzl_9c62bf5c-48e7-4250-ba5d-5e1a29efbe26/manager/0.log"
Nov 04 00:38:05.279534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:05.279494    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-w2cn8_9e5dacd3-94a5-4e52-bdb4-26f2fe871a14/manager/0.log"
Nov 04 00:38:06.030206 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:06.030171    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"]
Nov 04 00:38:06.032819 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:06.032784    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae575d-gts9n"]
Nov 04 00:38:07.018353 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:07.018305    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4" path="/var/lib/kubelet/pods/9c1ae2d4-7398-4db8-bb7b-bfa8c36ebba4/volumes"
Nov 04 00:38:16.029031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:16.028997    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"]
Nov 04 00:38:16.031526 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:16.031503    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f46a44-dq2pt"]
Nov 04 00:38:17.016541 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:17.016498    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be6b639-36f2-4530-888a-06cc6e0a2291" path="/var/lib/kubelet/pods/5be6b639-36f2-4530-888a-06cc6e0a2291/volumes"
Nov 04 00:38:17.167439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:17.167406    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 00:38:17.359131 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:17.359052    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 00:38:17.583783 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:17.583753    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 00:38:17.853050 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:17.853016    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-4cp2t_07f9e9ff-13f0-4d3c-ba08-a308ce67d227/manager/1.log"
Nov 04 00:38:18.076111 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:18.076089    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 00:38:20.027592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:20.027557    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"]
Nov 04 00:38:20.030005 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:20.029982    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce365dd-z8nzn"]
Nov 04 00:38:21.018644 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:21.018605    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058ec553-4725-43de-aeb2-08c4df5c4f56" path="/var/lib/kubelet/pods/058ec553-4725-43de-aeb2-08c4df5c4f56/volumes"
Nov 04 00:38:26.027037 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:26.026997    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"]
Nov 04 00:38:26.029553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:26.029525    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae99f35d-q9hhm"]
Nov 04 00:38:27.016012 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:27.015977    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc998e42-fbff-484a-9736-0ced8d7afec3" path="/var/lib/kubelet/pods/dc998e42-fbff-484a-9736-0ced8d7afec3/volumes"
Nov 04 00:38:28.021258 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:38:28.021222    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:38:28.501424 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:28.501394    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 00:38:29.547969 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:29.547929    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-4cp2t_07f9e9ff-13f0-4d3c-ba08-a308ce67d227/manager/0.log"
Nov 04 00:38:34.028003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:34.027965    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"]
Nov 04 00:38:34.030268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:34.030244    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f43b351-lth8f"]
Nov 04 00:38:35.016655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:35.016616    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1c2495-9587-48c0-ba35-2be588219891" path="/var/lib/kubelet/pods/1e1c2495-9587-48c0-ba35-2be588219891/volumes"
Nov 04 00:38:56.132075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:56.132030    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:38:56.154145 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:38:56.154112    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:38:58.022720 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:38:58.022674    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:39:01.847216 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:01.847183    5074 scope.go:122] "RemoveContainer" containerID="2ba0056154771bfecf9d91f4c4db18af09b798bc0dabf25f372e1b1f5bb4825c"
Nov 04 00:39:01.857009 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:01.856987    5074 scope.go:122] "RemoveContainer" containerID="b0f879181306758026bda3b860ffc1fa0e6e7d64321fdc0ff116b4fccaaa2bf5"
Nov 04 00:39:01.865809 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:01.865790    5074 scope.go:122] "RemoveContainer" containerID="6a4750097337aed7fd7b8b929b62c2a0d40cef3a479b0e87651a59264d69b45d"
Nov 04 00:39:10.027515 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:10.027477    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"]
Nov 04 00:39:10.030962 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:10.030934    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141690-mm7kl"]
Nov 04 00:39:11.016552 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.016520    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8780bcf-f7bd-428c-aa5e-94faf82ae061" path="/var/lib/kubelet/pods/f8780bcf-f7bd-428c-aa5e-94faf82ae061/volumes"
Nov 04 00:39:11.324474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.324293    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-4cp2t"]
Nov 04 00:39:11.325075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.324721    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager" containerID="cri-o://4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc" gracePeriod=30
Nov 04 00:39:11.595308 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.595282    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 04 00:39:11.631358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.631324    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs\") pod \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") "
Nov 04 00:39:11.631555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.631388    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn\" (UniqueName: \"kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn\") pod \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") "
Nov 04 00:39:11.631555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.631433    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert\") pod \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\" (UID: \"07f9e9ff-13f0-4d3c-ba08-a308ce67d227\") "
Nov 04 00:39:11.633512 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.633472    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs" pod "07f9e9ff-13f0-4d3c-ba08-a308ce67d227" (UID: "07f9e9ff-13f0-4d3c-ba08-a308ce67d227"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:39:11.633874 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.633847    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn" pod "07f9e9ff-13f0-4d3c-ba08-a308ce67d227" (UID: "07f9e9ff-13f0-4d3c-ba08-a308ce67d227"). InnerVolumeSpecName "kube-api-access-f8wpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:11.633991 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.633957    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert" pod "07f9e9ff-13f0-4d3c-ba08-a308ce67d227" (UID: "07f9e9ff-13f0-4d3c-ba08-a308ce67d227"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:39:11.732799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.732762    5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-metrics-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:11.732799 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.732800    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wpn\" (UniqueName: \"kubernetes.io/projected/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-kube-api-access-f8wpn\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:11.733065 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:11.732815    5074 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9ff-13f0-4d3c-ba08-a308ce67d227-webhook-cert\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:12.347117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.347076    5074 generic.go:356] "Generic (PLEG): container finished" podID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerID="4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc" exitCode=0
Nov 04 00:39:12.347626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.347129    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t"
Nov 04 00:39:12.347626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.347165    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" event={"ID":"07f9e9ff-13f0-4d3c-ba08-a308ce67d227","Type":"ContainerDied","Data":"4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc"}
Nov 04 00:39:12.347626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.347241    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-4cp2t" event={"ID":"07f9e9ff-13f0-4d3c-ba08-a308ce67d227","Type":"ContainerDied","Data":"c762c085cf446939efe1f0928742c50e904930292a6f9e8ec1660d75b742b461"}
Nov 04 00:39:12.347626 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.347281    5074 scope.go:122] "RemoveContainer" containerID="4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc"
Nov 04 00:39:12.358133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.358110    5074 scope.go:122] "RemoveContainer" containerID="27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"
Nov 04 00:39:12.370592 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.370560    5074 scope.go:122] "RemoveContainer" containerID="4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc"
Nov 04 00:39:12.370915 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:39:12.370892    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc\": container with ID starting with 4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc not found: ID does not exist" containerID="4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc"
Nov 04 00:39:12.370997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.370928    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc"} err="failed to get container status \"4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc\": rpc error: code = NotFound desc = could not find container \"4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc\": container with ID starting with 4757807412e19f07349eb899dde95fad8d1d871ef8a94666ea8303c7cb6ce5bc not found: ID does not exist"
Nov 04 00:39:12.370997 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.370950    5074 scope.go:122] "RemoveContainer" containerID="27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"
Nov 04 00:39:12.371213 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:39:12.371193    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47\": container with ID starting with 27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47 not found: ID does not exist" containerID="27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"
Nov 04 00:39:12.371268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.371218    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47"} err="failed to get container status \"27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47\": rpc error: code = NotFound desc = could not find container \"27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47\": container with ID starting with 27c7cca306fe7d98c0ac082e57151dbb71a93ae69bf0550e2ed5ca15d69fab47 not found: ID does not exist"
Nov 04 00:39:12.379555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.379516    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-4cp2t"]
Nov 04 00:39:12.379725 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:12.379584    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-4cp2t"]
Nov 04 00:39:13.020313 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:13.020266    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" path="/var/lib/kubelet/pods/07f9e9ff-13f0-4d3c-ba08-a308ce67d227/volumes"
Nov 04 00:39:20.027110 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:20.027069    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"]
Nov 04 00:39:20.029443 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:20.029419    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf83ed2-2wzw8"]
Nov 04 00:39:21.015937 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:21.015897    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a920b9a8-f55f-45b7-9468-02a1ed04a7b3" path="/var/lib/kubelet/pods/a920b9a8-f55f-45b7-9468-02a1ed04a7b3/volumes"
Nov 04 00:39:28.020543 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:39:28.020497    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:39:40.027517 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:40.027483    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"]
Nov 04 00:39:40.029764 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:40.029723    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc98973-697d6"]
Nov 04 00:39:41.016298 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:41.016266    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3762366b-4fdb-40ad-ab98-4241d08dc3cc" path="/var/lib/kubelet/pods/3762366b-4fdb-40ad-ab98-4241d08dc3cc/volumes"
Nov 04 00:39:48.166387 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166355    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-wktkm"]
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166777    5074 cpu_manager.go:414] "containerMap: removing container" podUID="36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" containerName="revoke-keys"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166788    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" containerName="revoke-keys"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166803    5074 cpu_manager.go:414] "containerMap: removing container" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166808    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166831    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166838    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166903    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166914    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager"
Nov 04 00:39:48.166921 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166921    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c631f8d7-c832-4e70-b291-4e8629e024a5" containerName="main"
Nov 04 00:39:48.167266 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.166931    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" containerName="revoke-keys"
Nov 04 00:39:48.170305 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.170287    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.177734 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.177479    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 04 00:39:48.182258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.182219    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 04 00:39:48.182258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.182247    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 04 00:39:48.182465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.182308    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ww46r\""
Nov 04 00:39:48.183716 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.183693    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-wktkm"]
Nov 04 00:39:48.296122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.296089    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac114d7f-26b3-43c7-b52e-c273efafdcb7-webhook-cert\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.296269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.296139    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac114d7f-26b3-43c7-b52e-c273efafdcb7-metrics-certs\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.296269 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.296211    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkqc\" (UniqueName: \"kubernetes.io/projected/ac114d7f-26b3-43c7-b52e-c273efafdcb7-kube-api-access-hrkqc\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.397239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.397209    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac114d7f-26b3-43c7-b52e-c273efafdcb7-webhook-cert\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.397415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.397259    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac114d7f-26b3-43c7-b52e-c273efafdcb7-metrics-certs\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.397415 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.397286    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkqc\" (UniqueName: \"kubernetes.io/projected/ac114d7f-26b3-43c7-b52e-c273efafdcb7-kube-api-access-hrkqc\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.399786 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.399760    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac114d7f-26b3-43c7-b52e-c273efafdcb7-metrics-certs\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.399900 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.399823    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac114d7f-26b3-43c7-b52e-c273efafdcb7-webhook-cert\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.410628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.410602    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkqc\" (UniqueName: \"kubernetes.io/projected/ac114d7f-26b3-43c7-b52e-c273efafdcb7-kube-api-access-hrkqc\") pod \"maas-controller-9cc8d5ff6-wktkm\" (UID: \"ac114d7f-26b3-43c7-b52e-c273efafdcb7\") " pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.484951 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.484928    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:48.763098 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.763066    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-wktkm"]
Nov 04 00:39:48.764815 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:39:48.764775    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac114d7f_26b3_43c7_b52e_c273efafdcb7.slice/crio-0ffcac8ab30f36765d2959396abd8a0ce40c6a7a68a35708dbe3226831bd8a7c WatchSource:0}: Error finding container 0ffcac8ab30f36765d2959396abd8a0ce40c6a7a68a35708dbe3226831bd8a7c: Status 404 returned error can't find the container with id 0ffcac8ab30f36765d2959396abd8a0ce40c6a7a68a35708dbe3226831bd8a7c
Nov 04 00:39:48.766176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:48.766159    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:39:49.591815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:49.591776    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm" event={"ID":"ac114d7f-26b3-43c7-b52e-c273efafdcb7","Type":"ContainerStarted","Data":"0ffcac8ab30f36765d2959396abd8a0ce40c6a7a68a35708dbe3226831bd8a7c"}
Nov 04 00:39:50.028972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:50.028926    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"]
Nov 04 00:39:50.031929 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:50.031903    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b1806-5bql5"]
Nov 04 00:39:50.599572 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:50.599535    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm" event={"ID":"ac114d7f-26b3-43c7-b52e-c273efafdcb7","Type":"ContainerStarted","Data":"cd82554133228d4569f938879338a9977b4b8c8b1e85b1d3af3d50184cb03f8c"}
Nov 04 00:39:50.600088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:50.599601    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:39:50.629031 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:50.628983    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm" podStartSLOduration=1.200201778 podStartE2EDuration="2.628970002s" totalImagesPullingTime="1.428768224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:39:50.626391462 +0000 UTC m=+4255.725991607" watchObservedRunningTime="2026-11-04 00:39:50.628970002 +0000 UTC m=+4255.728570146"
Nov 04 00:39:51.015844 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:39:51.015804    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce218f33-c0f0-428e-9f5b-e30bbe253b61" path="/var/lib/kubelet/pods/ce218f33-c0f0-428e-9f5b-e30bbe253b61/volumes"
Nov 04 00:39:58.020477 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:39:58.020439    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:40:01.608932 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:01.608899    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc8d5ff6-wktkm"
Nov 04 00:40:01.898379 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:01.898292    5074 scope.go:122] "RemoveContainer" containerID="a0751fa379b62fb831b6b86692bc7c7eb2d37f76d48642f5ab47331d2a3b477e"
Nov 04 00:40:01.907649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:01.907622    5074 scope.go:122] "RemoveContainer" containerID="a1831c64dc092e1cfcb1d69d60e50f2042117469090182994c06878ab08e87c5"
Nov 04 00:40:01.916120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:01.916099    5074 scope.go:122] "RemoveContainer" containerID="6e26b63cf155b46a1dc4272ddc18260ca9b1fe157cf7e621811a4aa15259856c"
Nov 04 00:40:01.924408 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:01.924388    5074 scope.go:122] "RemoveContainer" containerID="5ff4ae306a53cd668f428eb10525ccebbf39b9a1a14d0c4c498da4dd91626ba0"
Nov 04 00:40:01.932793 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:01.932771    5074 scope.go:122] "RemoveContainer" containerID="0c020bfa185d346bfe41900845fd1e1d33b63c851afffdbfb0e0edb369f8cfe5"
Nov 04 00:40:27.031217 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:27.031169    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"]
Nov 04 00:40:27.035083 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:27.035049    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c26a1a-zz8mk"]
Nov 04 00:40:28.094312 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:40:28.094275    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:40:29.021356 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:29.021323    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045116e7-557f-406c-987b-e2659b6014ec" path="/var/lib/kubelet/pods/045116e7-557f-406c-987b-e2659b6014ec/volumes"
Nov 04 00:40:37.027483 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:37.027434    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"]
Nov 04 00:40:37.029837 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:37.029812    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42672ff8-sscmz"]
Nov 04 00:40:39.015896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:39.015851    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e291f68-9ec7-4ffc-9a88-cee4d262fddf" path="/var/lib/kubelet/pods/8e291f68-9ec7-4ffc-9a88-cee4d262fddf/volumes"
Nov 04 00:40:49.029434 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:49.029398    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"]
Nov 04 00:40:49.032013 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:49.031989    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77929a-fvdp4"]
Nov 04 00:40:51.072126 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:51.072089    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274ddccb-879b-4a79-b92a-e3846efaabb7" path="/var/lib/kubelet/pods/274ddccb-879b-4a79-b92a-e3846efaabb7/volumes"
Nov 04 00:40:58.020949 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:40:58.020902    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:40:59.029090 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:59.029054    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"]
Nov 04 00:40:59.031214 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:40:59.031188    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa46684a-85l95"]
Nov 04 00:41:01.016678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:01.016636    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="238dd50d-e199-4f30-8238-b22a43370116" path="/var/lib/kubelet/pods/238dd50d-e199-4f30-8238-b22a43370116/volumes"
Nov 04 00:41:01.976372 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:01.976336    5074 scope.go:122] "RemoveContainer" containerID="ed392eb6d5224e1e3569751eb4fb0c1c175900c033ad9a5db1b73fe6cf6d423c"
Nov 04 00:41:01.988153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:01.988130    5074 scope.go:122] "RemoveContainer" containerID="87cc59d8b5e8b495b965f9dc3dc6bfde59a47fe89512d4b2bd9b5b737dae6bad"
Nov 04 00:41:01.996816 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:01.996795    5074 scope.go:122] "RemoveContainer" containerID="d39ee780162ad0fc497f24463009aa39b6660cbf0d8d0edb6061ae1f625a6bf0"
Nov 04 00:41:02.004789 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:02.004769    5074 scope.go:122] "RemoveContainer" containerID="e1a53cca36d476a268a6ac7aa3287c2a472a205acb2e7e4addc0d3989719c3a0"
Nov 04 00:41:09.072930 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:09.072888    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"]
Nov 04 00:41:09.075393 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:09.075365    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca73e355-cgkkm"]
Nov 04 00:41:11.016295 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:11.016258    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9eaedd-71e2-41a1-b61f-61c385c290c7" path="/var/lib/kubelet/pods/9d9eaedd-71e2-41a1-b61f-61c385c290c7/volumes"
Nov 04 00:41:28.021036 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:41:28.020987    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:41:31.027359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:31.027321    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"]
Nov 04 00:41:31.031846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:31.031813    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1421d2-72qmm"]
Nov 04 00:41:33.016880 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:33.016837    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221cf970-43a4-4aca-bef3-d3efe4d53398" path="/var/lib/kubelet/pods/221cf970-43a4-4aca-bef3-d3efe4d53398/volumes"
Nov 04 00:41:36.036025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:36.035937    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"]
Nov 04 00:41:36.040478 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:36.040446    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f906e62-6lmtw"]
Nov 04 00:41:37.016686 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:37.016647    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc07ee9-08a5-4d26-81af-dd06828fe786" path="/var/lib/kubelet/pods/8bc07ee9-08a5-4d26-81af-dd06828fe786/volumes"
Nov 04 00:41:48.028045 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:48.028004    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"]
Nov 04 00:41:48.030679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:48.030651    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c800c36a-lfzt5"]
Nov 04 00:41:49.016237 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:41:49.016201    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e2500a-38b3-4493-8ce1-84f2c82c01d6" path="/var/lib/kubelet/pods/13e2500a-38b3-4493-8ce1-84f2c82c01d6/volumes"
Nov 04 00:41:58.026079 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:41:58.026005    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:42:02.064765 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:02.064364    5074 scope.go:122] "RemoveContainer" containerID="5af308ed58e40eb9f52ca086308df614aafb6fe74b831284860c2af3416a1c4e"
Nov 04 00:42:02.082314 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:02.082293    5074 scope.go:122] "RemoveContainer" containerID="70a95e1d6d28c08c322bbcb65c2b6be8f1fa1c3b59a2d0f64694295b56c126a3"
Nov 04 00:42:02.092650 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:02.092627    5074 scope.go:122] "RemoveContainer" containerID="ee223b38a75589335628b089a7ed20a17266fb798d6f0a79dfc361686a4d3222"
Nov 04 00:42:02.131004 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:02.130952    5074 scope.go:122] "RemoveContainer" containerID="4166267c58757f9078cb847bda0a9a766554ea8304e24503251c2949e91a7e42"
Nov 04 00:42:17.091441 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.091388    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"]
Nov 04 00:42:17.091982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.091957    5074 cpu_manager.go:414] "containerMap: removing container" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager"
Nov 04 00:42:17.091982 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.091977    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f9e9ff-13f0-4d3c-ba08-a308ce67d227" containerName="manager"
Nov 04 00:42:17.095581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.095551    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.104054 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.104015    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:42:17.104215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.104127    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:42:17.106582 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.106545    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"]
Nov 04 00:42:17.229811 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.229758    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjgj\" (UniqueName: \"kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.230025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.229889    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.230025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.229938    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.331184 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.331141    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjgj\" (UniqueName: \"kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.331403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.331240    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.331403 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.331275    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.332122 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.332084    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.333888 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.333866    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.344867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.344781    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjgj\" (UniqueName: \"kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj\") pod \"maas-api-revoke-sub-4fb9147e-dcs24\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.411730 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.411683    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:17.694646 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:17.694611    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"]
Nov 04 00:42:17.697172 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:42:17.697132    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4793affc_5ab4_4c81_95d6_75dae0ac6763.slice/crio-18f59c9f299fba7fbd0034cf717871e3444299a179d518ddb1cc4a37503e4612 WatchSource:0}: Error finding container 18f59c9f299fba7fbd0034cf717871e3444299a179d518ddb1cc4a37503e4612: Status 404 returned error can't find the container with id 18f59c9f299fba7fbd0034cf717871e3444299a179d518ddb1cc4a37503e4612
Nov 04 00:42:18.600518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:18.600480    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24" event={"ID":"4793affc-5ab4-4c81-95d6-75dae0ac6763","Type":"ContainerStarted","Data":"238d52e977f34f201adcef2403c83fe0c7fffbdb337113aa5216ab8a582ea473"}
Nov 04 00:42:18.600972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:18.600550    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24" event={"ID":"4793affc-5ab4-4c81-95d6-75dae0ac6763","Type":"ContainerStarted","Data":"18f59c9f299fba7fbd0034cf717871e3444299a179d518ddb1cc4a37503e4612"}
Nov 04 00:42:18.620695 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:18.620645    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24" podStartSLOduration=1.6206327329999999 podStartE2EDuration="1.620632733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:42:18.620035261 +0000 UTC m=+4403.719635402" watchObservedRunningTime="2026-11-04 00:42:18.620632733 +0000 UTC m=+4403.720232909"
Nov 04 00:42:19.607253 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:19.607216    5074 generic.go:356] "Generic (PLEG): container finished" podID="4793affc-5ab4-4c81-95d6-75dae0ac6763" containerID="238d52e977f34f201adcef2403c83fe0c7fffbdb337113aa5216ab8a582ea473" exitCode=0
Nov 04 00:42:19.607678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:19.607303    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24" event={"ID":"4793affc-5ab4-4c81-95d6-75dae0ac6763","Type":"ContainerDied","Data":"238d52e977f34f201adcef2403c83fe0c7fffbdb337113aa5216ab8a582ea473"}
Nov 04 00:42:20.747333 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.747307    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:20.864476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.864403    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj\" (UniqueName: \"kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj\") pod \"4793affc-5ab4-4c81-95d6-75dae0ac6763\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") "
Nov 04 00:42:20.864476 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.864458    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca\") pod \"4793affc-5ab4-4c81-95d6-75dae0ac6763\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") "
Nov 04 00:42:20.864668 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.864540    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca\") pod \"4793affc-5ab4-4c81-95d6-75dae0ac6763\" (UID: \"4793affc-5ab4-4c81-95d6-75dae0ac6763\") "
Nov 04 00:42:20.864939 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.864905    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca" pod "4793affc-5ab4-4c81-95d6-75dae0ac6763" (UID: "4793affc-5ab4-4c81-95d6-75dae0ac6763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:42:20.866518 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.866492    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca" pod "4793affc-5ab4-4c81-95d6-75dae0ac6763" (UID: "4793affc-5ab4-4c81-95d6-75dae0ac6763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:42:20.866630 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.866579    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj" pod "4793affc-5ab4-4c81-95d6-75dae0ac6763" (UID: "4793affc-5ab4-4c81-95d6-75dae0ac6763"). InnerVolumeSpecName "kube-api-access-jvjgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:42:20.965892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.965860    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:42:20.965892 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.965888    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjgj\" (UniqueName: \"kubernetes.io/projected/4793affc-5ab4-4c81-95d6-75dae0ac6763-kube-api-access-jvjgj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:42:20.966067 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:20.965899    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4793affc-5ab4-4c81-95d6-75dae0ac6763-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:42:21.619033 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:21.618993    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24" event={"ID":"4793affc-5ab4-4c81-95d6-75dae0ac6763","Type":"ContainerDied","Data":"18f59c9f299fba7fbd0034cf717871e3444299a179d518ddb1cc4a37503e4612"}
Nov 04 00:42:21.619211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:21.619054    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f59c9f299fba7fbd0034cf717871e3444299a179d518ddb1cc4a37503e4612"
Nov 04 00:42:21.619211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:21.619010    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"
Nov 04 00:42:24.027153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:24.027113    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"]
Nov 04 00:42:24.029485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:24.029455    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787c5223-mkqdp"]
Nov 04 00:42:25.016407 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:25.016377    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b982ee71-bf94-4dbf-a23a-c845080d96eb" path="/var/lib/kubelet/pods/b982ee71-bf94-4dbf-a23a-c845080d96eb/volumes"
Nov 04 00:42:28.020468 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:42:28.020421    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:42:36.028058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:36.028012    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"]
Nov 04 00:42:36.030620 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:36.030578    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ddb2ad-4cqn5"]
Nov 04 00:42:37.016370 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:37.016332    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ac2b6e-1c6d-4abe-894f-2f45ac1492b2" path="/var/lib/kubelet/pods/36ac2b6e-1c6d-4abe-894f-2f45ac1492b2/volumes"
Nov 04 00:42:42.318400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.317873    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:42:42.318400 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.317967    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:42:42.927492 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.927448    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"]
Nov 04 00:42:42.928117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.928095    5074 cpu_manager.go:414] "containerMap: removing container" podUID="4793affc-5ab4-4c81-95d6-75dae0ac6763" containerName="revoke-keys"
Nov 04 00:42:42.928212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.928121    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4793affc-5ab4-4c81-95d6-75dae0ac6763" containerName="revoke-keys"
Nov 04 00:42:42.928286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.928274    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4793affc-5ab4-4c81-95d6-75dae0ac6763" containerName="revoke-keys"
Nov 04 00:42:42.931952 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.931925    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:42.942291 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.942264    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"]
Nov 04 00:42:42.942664 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.942643    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:42:42.942866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:42.942842    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:42:43.099303 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.099263    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.099763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.099714    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlfj\" (UniqueName: \"kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.100089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.100067    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.200854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.200812    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlfj\" (UniqueName: \"kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.200854 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.200870    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.201171 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.200935    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.201601 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.201579    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.203760 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.203722    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.215045 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.215017    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlfj\" (UniqueName: \"kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj\") pod \"maas-api-revoke-sub-b1e0db8e-l675n\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.257678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.257640    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:43.554384 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.554288    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"]
Nov 04 00:42:43.558712 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:42:43.558677    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbaf8d2_0891_423f_ae53_ce380fb8e0d8.slice/crio-bd7f2e092d04cd9ea4e2de386db688be87904c1abc8de2d9f5825da37e7e2ee5 WatchSource:0}: Error finding container bd7f2e092d04cd9ea4e2de386db688be87904c1abc8de2d9f5825da37e7e2ee5: Status 404 returned error can't find the container with id bd7f2e092d04cd9ea4e2de386db688be87904c1abc8de2d9f5825da37e7e2ee5
Nov 04 00:42:43.760089 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:43.760047    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n" event={"ID":"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8","Type":"ContainerStarted","Data":"bd7f2e092d04cd9ea4e2de386db688be87904c1abc8de2d9f5825da37e7e2ee5"}
Nov 04 00:42:44.788715 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:44.788624    5074 generic.go:356] "Generic (PLEG): container finished" podID="fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" containerID="c2973b446d34d2f012c9029b07e67ecdcd924b77c4188e06fc0b1675ab4bd456" exitCode=0
Nov 04 00:42:44.790554 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:44.790521    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n" event={"ID":"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8","Type":"ContainerDied","Data":"c2973b446d34d2f012c9029b07e67ecdcd924b77c4188e06fc0b1675ab4bd456"}
Nov 04 00:42:45.965516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:45.965486    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:46.075046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.074970    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca\") pod \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") "
Nov 04 00:42:46.075046 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.075027    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca\") pod \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") "
Nov 04 00:42:46.075293 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.075062    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj\" (UniqueName: \"kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj\") pod \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\" (UID: \"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8\") "
Nov 04 00:42:46.075411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.075387    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca" pod "fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" (UID: "fdbaf8d2-0891-423f-ae53-ce380fb8e0d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:42:46.077146 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.077114    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca" pod "fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" (UID: "fdbaf8d2-0891-423f-ae53-ce380fb8e0d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:42:46.077270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.077221    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj" pod "fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" (UID: "fdbaf8d2-0891-423f-ae53-ce380fb8e0d8"). InnerVolumeSpecName "kube-api-access-rqlfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:42:46.176719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.176687    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:42:46.176719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.176716    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:42:46.176926 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.176726    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlfj\" (UniqueName: \"kubernetes.io/projected/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8-kube-api-access-rqlfj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:42:46.801268 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.801233    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"
Nov 04 00:42:46.801471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.801272    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n" event={"ID":"fdbaf8d2-0891-423f-ae53-ce380fb8e0d8","Type":"ContainerDied","Data":"bd7f2e092d04cd9ea4e2de386db688be87904c1abc8de2d9f5825da37e7e2ee5"}
Nov 04 00:42:46.801471 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:42:46.801329    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7f2e092d04cd9ea4e2de386db688be87904c1abc8de2d9f5825da37e7e2ee5"
Nov 04 00:42:58.020723 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:42:58.020685    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:43:02.235712 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:43:02.235678    5074 scope.go:122] "RemoveContainer" containerID="8b14895b44852180aef7688f2816f86a15d3d2a9eb99f99a1468d31a3bc36e68"
Nov 04 00:43:02.244943 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:43:02.244919    5074 scope.go:122] "RemoveContainer" containerID="df40ac677726a18e89c3a8c76abf459f00add9c9e53255d41b4ecc886181d070"
Nov 04 00:43:02.252834 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:43:02.252812    5074 scope.go:122] "RemoveContainer" containerID="d788af6c6f696b29324a21c8984af9a8c2e54b3ffdba40744efe5c3c551d6d2e"
Nov 04 00:43:02.260714 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:43:02.260691    5074 scope.go:122] "RemoveContainer" containerID="fb7a67d19b11e5ff83dc1702d01d708d6547404de5912a58ce62c20001e9cf4d"
Nov 04 00:43:28.021655 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:43:28.021619    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:43:56.175120 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:43:56.175083    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:43:56.195354 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:43:56.195321    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:43:58.021942 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:43:58.021896    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:44:02.292891 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:44:02.292860    5074 scope.go:122] "RemoveContainer" containerID="8486396b39f081125e46c2012eb06f8c64d14cb70bbf337fe24b8e5b3d93d42b"
Nov 04 00:44:02.302056 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:44:02.302039    5074 scope.go:122] "RemoveContainer" containerID="eda2f3bf2d82d0bf8a2dcca1a0e35296cce7d258e542c372146b080829e35556"
Nov 04 00:44:28.022810 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:44:28.022766    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:44:32.428692 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:44:32.428649    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" podUID="14f459d1824c1ea630aa90a99306c647" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:44:58.026932 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:44:58.026842    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:45:00.145720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.145685    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"]
Nov 04 00:45:00.146195 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.146174    5074 cpu_manager.go:414] "containerMap: removing container" podUID="fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" containerName="revoke-keys"
Nov 04 00:45:00.146271 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.146195    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" containerName="revoke-keys"
Nov 04 00:45:00.146340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.146327    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" containerName="revoke-keys"
Nov 04 00:45:00.149456 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.149439    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.157452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.157426    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:45:00.159822 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.159800    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"]
Nov 04 00:45:00.310762 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.310690    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wpwd\" (UniqueName: \"kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.310941 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.310877    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.412073 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.411972    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.412263 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.412089    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wpwd\" (UniqueName: \"kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.412680 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.412657    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.425411 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.425377    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wpwd\" (UniqueName: \"kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.464338 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.464303    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:00.747640 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:00.747608    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"]
Nov 04 00:45:00.750018 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:45:00.749980    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8906987_4563_434b_b722_215b9247bb6c.slice/crio-c989d0c02852ad2821d7b9a4bf4e91f3e8234bd6faccf8333b9c368a68fb0f69 WatchSource:0}: Error finding container c989d0c02852ad2821d7b9a4bf4e91f3e8234bd6faccf8333b9c368a68fb0f69: Status 404 returned error can't find the container with id c989d0c02852ad2821d7b9a4bf4e91f3e8234bd6faccf8333b9c368a68fb0f69
Nov 04 00:45:01.540895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:01.540859    5074 generic.go:356] "Generic (PLEG): container finished" podID="e8906987-4563-434b-b722-215b9247bb6c" containerID="b9015f093033803957ff6fe1582bad0493abcfafa8cb02c0110df12e9a6a3d32" exitCode=7
Nov 04 00:45:01.541425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:01.540930    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z" event={"ID":"e8906987-4563-434b-b722-215b9247bb6c","Type":"ContainerDied","Data":"b9015f093033803957ff6fe1582bad0493abcfafa8cb02c0110df12e9a6a3d32"}
Nov 04 00:45:01.541425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:01.540971    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z" event={"ID":"e8906987-4563-434b-b722-215b9247bb6c","Type":"ContainerStarted","Data":"c989d0c02852ad2821d7b9a4bf4e91f3e8234bd6faccf8333b9c368a68fb0f69"}
Nov 04 00:45:01.541425 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:01.541208    5074 scope.go:122] "RemoveContainer" containerID="b9015f093033803957ff6fe1582bad0493abcfafa8cb02c0110df12e9a6a3d32"
Nov 04 00:45:02.550015 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:02.549923    5074 generic.go:356] "Generic (PLEG): container finished" podID="e8906987-4563-434b-b722-215b9247bb6c" containerID="c1214312ae38e8dc0cf29c05e96c80bcb4645db8578db8b8b8d78faa06d400e6" exitCode=7
Nov 04 00:45:02.550015 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:02.549989    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z" event={"ID":"e8906987-4563-434b-b722-215b9247bb6c","Type":"ContainerDied","Data":"c1214312ae38e8dc0cf29c05e96c80bcb4645db8578db8b8b8d78faa06d400e6"}
Nov 04 00:45:02.550465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:02.550061    5074 scope.go:122] "RemoveContainer" containerID="b9015f093033803957ff6fe1582bad0493abcfafa8cb02c0110df12e9a6a3d32"
Nov 04 00:45:02.550465 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:02.550289    5074 scope.go:122] "RemoveContainer" containerID="c1214312ae38e8dc0cf29c05e96c80bcb4645db8578db8b8b8d78faa06d400e6"
Nov 04 00:45:02.550571 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:45:02.550546    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z_odh-ai-gateway-infra(e8906987-4563-434b-b722-215b9247bb6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z" podUID="e8906987-4563-434b-b722-215b9247bb6c"
Nov 04 00:45:03.558188 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:03.558153    5074 scope.go:122] "RemoveContainer" containerID="c1214312ae38e8dc0cf29c05e96c80bcb4645db8578db8b8b8d78faa06d400e6"
Nov 04 00:45:03.558771 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:45:03.558394    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z_odh-ai-gateway-infra(e8906987-4563-434b-b722-215b9247bb6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z" podUID="e8906987-4563-434b-b722-215b9247bb6c"
Nov 04 00:45:14.744664 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:14.744628    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"]
Nov 04 00:45:14.924733 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:14.924701    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:14.952818 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:14.952789    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca\") pod \"e8906987-4563-434b-b722-215b9247bb6c\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") "
Nov 04 00:45:14.952974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:14.952880    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd\" (UniqueName: \"kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd\") pod \"e8906987-4563-434b-b722-215b9247bb6c\" (UID: \"e8906987-4563-434b-b722-215b9247bb6c\") "
Nov 04 00:45:14.953255 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:14.953228    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca" pod "e8906987-4563-434b-b722-215b9247bb6c" (UID: "e8906987-4563-434b-b722-215b9247bb6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:45:14.955025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:14.954997    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd" pod "e8906987-4563-434b-b722-215b9247bb6c" (UID: "e8906987-4563-434b-b722-215b9247bb6c"). InnerVolumeSpecName "kube-api-access-4wpwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:45:15.054074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.054037    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wpwd\" (UniqueName: \"kubernetes.io/projected/e8906987-4563-434b-b722-215b9247bb6c-kube-api-access-4wpwd\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:15.054074 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.054071    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8906987-4563-434b-b722-215b9247bb6c-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:15.640655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.640230    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z" event={"ID":"e8906987-4563-434b-b722-215b9247bb6c","Type":"ContainerDied","Data":"c989d0c02852ad2821d7b9a4bf4e91f3e8234bd6faccf8333b9c368a68fb0f69"}
Nov 04 00:45:15.640655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.640310    5074 scope.go:122] "RemoveContainer" containerID="c1214312ae38e8dc0cf29c05e96c80bcb4645db8578db8b8b8d78faa06d400e6"
Nov 04 00:45:15.640655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.640454    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"
Nov 04 00:45:15.668082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.668048    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"]
Nov 04 00:45:15.671972 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:15.671943    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ca4fbe-2989588lpc5z"]
Nov 04 00:45:17.017634 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:17.017597    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8906987-4563-434b-b722-215b9247bb6c" path="/var/lib/kubelet/pods/e8906987-4563-434b-b722-215b9247bb6c/volumes"
Nov 04 00:45:28.021026 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:45:28.020985    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:45:42.308575 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:42.308523    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:45:42.309377 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:42.309341    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:45:57.316901 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:57.316849    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:45:57.317516 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:45:57.317481    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:45:58.021564 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:45:58.021517    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:46:08.696980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:46:08.696872    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nlkqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:46:08.696980 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:46:08.696956    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nlkqs" podUID="060dc557-cd0d-437a-b84e-6e52bb4e79a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:46:12.315994 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:46:12.315916    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m6wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:46:12.316689 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:46:12.316651    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m6wnb" podUID="eda3869d-7ca3-4325-b816-dd5e5153ba6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:46:28.021054 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:46:28.021011    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:46:34.059359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:46:34.058840    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:46:34.059359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:46:34.058931    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:46:58.021853 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:46:58.021807    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:47:21.028284 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:47:21.028235    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"]
Nov 04 00:47:21.030976 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:47:21.030952    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb9147e-dcs24"]
Nov 04 00:47:23.016524 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:47:23.016491    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4793affc-5ab4-4c81-95d6-75dae0ac6763" path="/var/lib/kubelet/pods/4793affc-5ab4-4c81-95d6-75dae0ac6763/volumes"
Nov 04 00:47:28.021752 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:47:28.021692    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:47:47.027638 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:47:47.027598    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"]
Nov 04 00:47:47.029992 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:47:47.029895    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e0db8e-l675n"]
Nov 04 00:47:49.021571 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:47:49.021530    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbaf8d2-0891-423f-ae53-ce380fb8e0d8" path="/var/lib/kubelet/pods/fdbaf8d2-0891-423f-ae53-ce380fb8e0d8/volumes"
Nov 04 00:47:58.021049 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:47:58.021008    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:48:28.020573 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:48:28.020524    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:48:56.212678 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:48:56.212637    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:48:56.237501 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:48:56.237459    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:48:58.021222 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:48:58.021186    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:49:02.368036 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:49:02.368005    5074 scope.go:122] "RemoveContainer" containerID="c2973b446d34d2f012c9029b07e67ecdcd924b77c4188e06fc0b1675ab4bd456"
Nov 04 00:49:02.376978 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:49:02.376952    5074 scope.go:122] "RemoveContainer" containerID="238d52e977f34f201adcef2403c83fe0c7fffbdb337113aa5216ab8a582ea473"
Nov 04 00:49:28.020961 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:49:28.020919    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:49:58.023040 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:49:58.022999    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:50:28.021235 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:50:28.021193    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:50:58.070446 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:50:58.070223    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:51:28.021879 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:51:28.021839    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:51:58.024243 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:51:58.024198    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:52:28.020442 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:52:28.020404    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:52:58.136119 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:52:58.136080    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:53:28.020589 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:53:28.020549    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:53:56.249469 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:53:56.249378    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:53:56.280481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:53:56.280442    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:53:58.022298 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:53:58.022257    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:54:28.020324 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:54:28.020276    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:54:58.022154 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:54:58.022107    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:55:04.411611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.411568    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"]
Nov 04 00:55:04.412270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.412103    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e8906987-4563-434b-b722-215b9247bb6c" containerName="cleanup"
Nov 04 00:55:04.412270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.412123    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8906987-4563-434b-b722-215b9247bb6c" containerName="cleanup"
Nov 04 00:55:04.412270 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.412230    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8906987-4563-434b-b722-215b9247bb6c" containerName="cleanup"
Nov 04 00:55:04.415543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.415519    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.428660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.428517    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b34223\"/\"openshift-service-ca.crt\""
Nov 04 00:55:04.428660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.428630    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b34223\"/\"default-dockercfg-2lft6\""
Nov 04 00:55:04.428660 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.428634    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b34223\"/\"kube-root-ca.crt\""
Nov 04 00:55:04.429133 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.429104    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b34223\"/\"simulator-single-main-b34223-kserve-self-signed-certs\""
Nov 04 00:55:04.431995 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.431949    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"]
Nov 04 00:55:04.454082 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.454046    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pghvr\" (UniqueName: \"kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.454285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.454118    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.454285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.454183    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.454285 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.454263    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.454475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.454297    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.454475 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.454385    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555158    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555196 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555208    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555249    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555296    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pghvr\" (UniqueName: \"kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555365    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555499 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555386    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555634    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555751 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555687    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.555950 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.555928    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.557697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.557673    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.558025 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.558006    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.567971 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.567950    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pghvr\" (UniqueName: \"kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr\") pod \"simulator-single-main-b34223-kserve-5d69b8966d-dx5fw\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") " pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:04.736359 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:04.736319    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:05.020114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:05.020084    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"]
Nov 04 00:55:05.021982 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:55:05.021952    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e626a7_9d22_4381_95a7_328f44f088d1.slice/crio-3d546edab16c51bf0bef0211aad850d25ffde2ef33ce8b55875eb5b15ae3c352 WatchSource:0}: Error finding container 3d546edab16c51bf0bef0211aad850d25ffde2ef33ce8b55875eb5b15ae3c352: Status 404 returned error can't find the container with id 3d546edab16c51bf0bef0211aad850d25ffde2ef33ce8b55875eb5b15ae3c352
Nov 04 00:55:05.103815 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:05.103781    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" event={"ID":"d9e626a7-9d22-4381-95a7-328f44f088d1","Type":"ContainerStarted","Data":"3d546edab16c51bf0bef0211aad850d25ffde2ef33ce8b55875eb5b15ae3c352"}
Nov 04 00:55:06.112309 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:06.112277    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" event={"ID":"d9e626a7-9d22-4381-95a7-328f44f088d1","Type":"ContainerStarted","Data":"3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39"}
Nov 04 00:55:06.133948 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:06.133883    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" podStartSLOduration=2.133868768 podStartE2EDuration="2.133868768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:55:06.132318177 +0000 UTC m=+5171.231918322" watchObservedRunningTime="2026-11-04 00:55:06.133868768 +0000 UTC m=+5171.233468909"
Nov 04 00:55:06.736720 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:06.736684    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:14.737328 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:14.737282    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:14.749923 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:14.749895    5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:15.173522 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:15.173433    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:55:28.021499 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:55:28.021460    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:55:44.039944 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.039910    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"]
Nov 04 00:55:44.040439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.040417    5074 cpu_manager.go:414] "containerMap: removing container" podUID="e8906987-4563-434b-b722-215b9247bb6c" containerName="cleanup"
Nov 04 00:55:44.040490 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.040442    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8906987-4563-434b-b722-215b9247bb6c" containerName="cleanup"
Nov 04 00:55:44.040573 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.040562    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8906987-4563-434b-b722-215b9247bb6c" containerName="cleanup"
Nov 04 00:55:44.043690 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.043673    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.050436 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.050410    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:55:44.052505 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.052308    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:55:44.069055 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.069020    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"]
Nov 04 00:55:44.153022 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.152993    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkt5m\" (UniqueName: \"kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.153176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.153053    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.153176 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.153150    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.254621 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.254588    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkt5m\" (UniqueName: \"kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.254826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.254648    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.254826 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.254676    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.255336 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.255316    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.257144 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.257121    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.267474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.267447    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkt5m\" (UniqueName: \"kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m\") pod \"maas-api-revoke-sub-2b45680a-m78vt\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.365956 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.365872    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:44.647867 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:44.647772    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"]
Nov 04 00:55:44.651295 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:55:44.651259    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22842901_61fb_47ff_9d3e_96eae1d7bbb5.slice/crio-0d19d31e1e56887a5338450d38576f4328d45fc9a4047b013800208424a07ca3 WatchSource:0}: Error finding container 0d19d31e1e56887a5338450d38576f4328d45fc9a4047b013800208424a07ca3: Status 404 returned error can't find the container with id 0d19d31e1e56887a5338450d38576f4328d45fc9a4047b013800208424a07ca3
Nov 04 00:55:45.332105 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:45.332070    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt" event={"ID":"22842901-61fb-47ff-9d3e-96eae1d7bbb5","Type":"ContainerStarted","Data":"0b8f3e3bfb6b28ef34e8feae9933edb3465dcddbfbbf6a51a4ac745232b10dfd"}
Nov 04 00:55:45.332553 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:45.332127    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt" event={"ID":"22842901-61fb-47ff-9d3e-96eae1d7bbb5","Type":"ContainerStarted","Data":"0d19d31e1e56887a5338450d38576f4328d45fc9a4047b013800208424a07ca3"}
Nov 04 00:55:45.352203 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:45.352157    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt" podStartSLOduration=1.352144685 podStartE2EDuration="1.352144685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:55:45.350918207 +0000 UTC m=+5210.450518365" watchObservedRunningTime="2026-11-04 00:55:45.352144685 +0000 UTC m=+5210.451744828"
Nov 04 00:55:46.338847 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:46.338809    5074 generic.go:356] "Generic (PLEG): container finished" podID="22842901-61fb-47ff-9d3e-96eae1d7bbb5" containerID="0b8f3e3bfb6b28ef34e8feae9933edb3465dcddbfbbf6a51a4ac745232b10dfd" exitCode=0
Nov 04 00:55:46.339326 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:46.338894    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt" event={"ID":"22842901-61fb-47ff-9d3e-96eae1d7bbb5","Type":"ContainerDied","Data":"0b8f3e3bfb6b28ef34e8feae9933edb3465dcddbfbbf6a51a4ac745232b10dfd"}
Nov 04 00:55:47.490719 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.490691    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:47.589398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.589310    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca\") pod \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") "
Nov 04 00:55:47.589398 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.589378    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca\") pod \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") "
Nov 04 00:55:47.589619 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.589416    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m\" (UniqueName: \"kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m\") pod \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\" (UID: \"22842901-61fb-47ff-9d3e-96eae1d7bbb5\") "
Nov 04 00:55:47.589866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.589818    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca" pod "22842901-61fb-47ff-9d3e-96eae1d7bbb5" (UID: "22842901-61fb-47ff-9d3e-96eae1d7bbb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:55:47.591681 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.591651    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca" pod "22842901-61fb-47ff-9d3e-96eae1d7bbb5" (UID: "22842901-61fb-47ff-9d3e-96eae1d7bbb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:55:47.591807 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.591773    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m" pod "22842901-61fb-47ff-9d3e-96eae1d7bbb5" (UID: "22842901-61fb-47ff-9d3e-96eae1d7bbb5"). InnerVolumeSpecName "kube-api-access-gkt5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:55:47.690296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.690256    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:55:47.690296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.690291    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22842901-61fb-47ff-9d3e-96eae1d7bbb5-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:55:47.690296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:47.690302    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkt5m\" (UniqueName: \"kubernetes.io/projected/22842901-61fb-47ff-9d3e-96eae1d7bbb5-kube-api-access-gkt5m\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:55:48.350485 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:48.350448    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt" event={"ID":"22842901-61fb-47ff-9d3e-96eae1d7bbb5","Type":"ContainerDied","Data":"0d19d31e1e56887a5338450d38576f4328d45fc9a4047b013800208424a07ca3"}
Nov 04 00:55:48.350662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:48.350496    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d19d31e1e56887a5338450d38576f4328d45fc9a4047b013800208424a07ca3"
Nov 04 00:55:48.350662 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:48.350462    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"
Nov 04 00:55:58.021891 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:55:58.021851    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:55:59.912905 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:55:59.912866    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"]
Nov 04 00:56:14.933895 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:14.933851    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" podUID="d9e626a7-9d22-4381-95a7-328f44f088d1" containerName="main" containerID="cri-o://3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39" gracePeriod=45
Nov 04 00:56:15.197870 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.197812    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:56:15.256896 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.256857    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir\") pod \"d9e626a7-9d22-4381-95a7-328f44f088d1\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") "
Nov 04 00:56:15.257092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.256939    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home\") pod \"d9e626a7-9d22-4381-95a7-328f44f088d1\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") "
Nov 04 00:56:15.257092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257002    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm\") pod \"d9e626a7-9d22-4381-95a7-328f44f088d1\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") "
Nov 04 00:56:15.257092 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257073    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache\") pod \"d9e626a7-9d22-4381-95a7-328f44f088d1\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") "
Nov 04 00:56:15.257294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257132    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs\") pod \"d9e626a7-9d22-4381-95a7-328f44f088d1\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") "
Nov 04 00:56:15.257294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257166    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr\" (UniqueName: \"kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr\") pod \"d9e626a7-9d22-4381-95a7-328f44f088d1\" (UID: \"d9e626a7-9d22-4381-95a7-328f44f088d1\") "
Nov 04 00:56:15.257294 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257166    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir" pod "d9e626a7-9d22-4381-95a7-328f44f088d1" (UID: "d9e626a7-9d22-4381-95a7-328f44f088d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:56:15.257474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257367    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache" pod "d9e626a7-9d22-4381-95a7-328f44f088d1" (UID: "d9e626a7-9d22-4381-95a7-328f44f088d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:56:15.257474 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257173    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home" pod "d9e626a7-9d22-4381-95a7-328f44f088d1" (UID: "d9e626a7-9d22-4381-95a7-328f44f088d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:56:15.257613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257587    5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-tmp-dir\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:15.257613 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257614    5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-home\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:15.257869 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.257629    5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-model-cache\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:15.259628 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.259597    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs" pod "d9e626a7-9d22-4381-95a7-328f44f088d1" (UID: "d9e626a7-9d22-4381-95a7-328f44f088d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:56:15.259726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.259688    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm" pod "d9e626a7-9d22-4381-95a7-328f44f088d1" (UID: "d9e626a7-9d22-4381-95a7-328f44f088d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:56:15.259726 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.259706    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr" pod "d9e626a7-9d22-4381-95a7-328f44f088d1" (UID: "d9e626a7-9d22-4381-95a7-328f44f088d1"). InnerVolumeSpecName "kube-api-access-pghvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:56:15.358282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.358249    5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e626a7-9d22-4381-95a7-328f44f088d1-dshm\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:15.358282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.358276    5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e626a7-9d22-4381-95a7-328f44f088d1-tls-certs\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:15.358282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.358287    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pghvr\" (UniqueName: \"kubernetes.io/projected/d9e626a7-9d22-4381-95a7-328f44f088d1-kube-api-access-pghvr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:15.500215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.500128    5074 generic.go:356] "Generic (PLEG): container finished" podID="d9e626a7-9d22-4381-95a7-328f44f088d1" containerID="3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39" exitCode=0
Nov 04 00:56:15.500215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.500167    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"
Nov 04 00:56:15.500215 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.500188    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" event={"ID":"d9e626a7-9d22-4381-95a7-328f44f088d1","Type":"ContainerDied","Data":"3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39"}
Nov 04 00:56:15.500520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.500225    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" event={"ID":"d9e626a7-9d22-4381-95a7-328f44f088d1","Type":"ContainerDied","Data":"3d546edab16c51bf0bef0211aad850d25ffde2ef33ce8b55875eb5b15ae3c352"}
Nov 04 00:56:15.500520 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.500248    5074 scope.go:122] "RemoveContainer" containerID="3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39"
Nov 04 00:56:15.510344 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.510325    5074 scope.go:122] "RemoveContainer" containerID="3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39"
Nov 04 00:56:15.510618 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:56:15.510597    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39\": container with ID starting with 3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39 not found: ID does not exist" containerID="3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39"
Nov 04 00:56:15.510671 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.510627    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39"} err="failed to get container status \"3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39\": rpc error: code = NotFound desc = could not find container \"3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39\": container with ID starting with 3682a0b1388fd4a8918efa6d11f626bb155d987b8ddde0dde2c07861bff09e39 not found: ID does not exist"
Nov 04 00:56:15.526315 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.526293    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"]
Nov 04 00:56:15.529918 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:15.529896    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw"]
Nov 04 00:56:16.162842 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:16.162798    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-b34223/simulator-single-main-b34223-kserve-5d69b8966d-dx5fw" podUID="d9e626a7-9d22-4381-95a7-328f44f088d1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.197:8000/ready\": context deadline exceeded"
Nov 04 00:56:17.016430 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:17.016392    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e626a7-9d22-4381-95a7-328f44f088d1" path="/var/lib/kubelet/pods/d9e626a7-9d22-4381-95a7-328f44f088d1/volumes"
Nov 04 00:56:28.020417 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:56:28.020382    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:56:43.399709 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.399617    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"]
Nov 04 00:56:43.400347 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.400322    5074 cpu_manager.go:414] "containerMap: removing container" podUID="22842901-61fb-47ff-9d3e-96eae1d7bbb5" containerName="revoke-keys"
Nov 04 00:56:43.400437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.400350    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22842901-61fb-47ff-9d3e-96eae1d7bbb5" containerName="revoke-keys"
Nov 04 00:56:43.400437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.400384    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d9e626a7-9d22-4381-95a7-328f44f088d1" containerName="main"
Nov 04 00:56:43.400437 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.400393    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e626a7-9d22-4381-95a7-328f44f088d1" containerName="main"
Nov 04 00:56:43.400611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.400506    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e626a7-9d22-4381-95a7-328f44f088d1" containerName="main"
Nov 04 00:56:43.400611 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.400527    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22842901-61fb-47ff-9d3e-96eae1d7bbb5" containerName="revoke-keys"
Nov 04 00:56:43.403864 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.403842    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.408780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.408681    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:56:43.411261 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.410697    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:56:43.418960 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.418928    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"]
Nov 04 00:56:43.536519 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.536483    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.536706 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.536625    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnxj\" (UniqueName: \"kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.536776 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.536696    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.637884 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.637844    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.638088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.637927    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.638088 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.638014    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnxj\" (UniqueName: \"kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.638709 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.638689    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.640254 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.640231    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.651609 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.651546    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnxj\" (UniqueName: \"kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.720535 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.720506    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:43.995127 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:43.995102    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"]
Nov 04 00:56:43.997321 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:56:43.997291    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a02bca_ee13_48ed_90bd_a0b2e19373e5.slice/crio-d7d4b5856024a993ea7d055ea671a650979cd7e5da92aace0d69716be018c6d9 WatchSource:0}: Error finding container d7d4b5856024a993ea7d055ea671a650979cd7e5da92aace0d69716be018c6d9: Status 404 returned error can't find the container with id d7d4b5856024a993ea7d055ea671a650979cd7e5da92aace0d69716be018c6d9
Nov 04 00:56:44.656296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:44.656209    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc" event={"ID":"03a02bca-ee13-48ed-90bd-a0b2e19373e5","Type":"ContainerStarted","Data":"cd965a491a70d4f2b113dfe47c0e780934c1f75c7de3f75e6ff4abe070488a4f"}
Nov 04 00:56:44.656296 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:44.656274    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc" event={"ID":"03a02bca-ee13-48ed-90bd-a0b2e19373e5","Type":"ContainerStarted","Data":"d7d4b5856024a993ea7d055ea671a650979cd7e5da92aace0d69716be018c6d9"}
Nov 04 00:56:44.676362 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:44.676314    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc" podStartSLOduration=1.6762996220000002 podStartE2EDuration="1.676299622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:44.675364448 +0000 UTC m=+5269.774964594" watchObservedRunningTime="2026-11-04 00:56:44.676299622 +0000 UTC m=+5269.775899767"
Nov 04 00:56:45.662866 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:45.662828    5074 generic.go:356] "Generic (PLEG): container finished" podID="03a02bca-ee13-48ed-90bd-a0b2e19373e5" containerID="cd965a491a70d4f2b113dfe47c0e780934c1f75c7de3f75e6ff4abe070488a4f" exitCode=0
Nov 04 00:56:45.663301 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:45.662873    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc" event={"ID":"03a02bca-ee13-48ed-90bd-a0b2e19373e5","Type":"ContainerDied","Data":"cd965a491a70d4f2b113dfe47c0e780934c1f75c7de3f75e6ff4abe070488a4f"}
Nov 04 00:56:46.815075 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.815046    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:46.970153 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.970120    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca\") pod \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") "
Nov 04 00:56:46.970332 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.970167    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj\" (UniqueName: \"kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj\") pod \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") "
Nov 04 00:56:46.970382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.970361    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca\") pod \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\" (UID: \"03a02bca-ee13-48ed-90bd-a0b2e19373e5\") "
Nov 04 00:56:46.970721 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.970697    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca" pod "03a02bca-ee13-48ed-90bd-a0b2e19373e5" (UID: "03a02bca-ee13-48ed-90bd-a0b2e19373e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:56:46.972232 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.972204    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca" pod "03a02bca-ee13-48ed-90bd-a0b2e19373e5" (UID: "03a02bca-ee13-48ed-90bd-a0b2e19373e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:56:46.972481 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:46.972440    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj" pod "03a02bca-ee13-48ed-90bd-a0b2e19373e5" (UID: "03a02bca-ee13-48ed-90bd-a0b2e19373e5"). InnerVolumeSpecName "kube-api-access-2bnxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:56:47.071846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:47.071810    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:47.071846 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:47.071843    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnxj\" (UniqueName: \"kubernetes.io/projected/03a02bca-ee13-48ed-90bd-a0b2e19373e5-kube-api-access-2bnxj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:47.072018 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:47.071860    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a02bca-ee13-48ed-90bd-a0b2e19373e5-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:47.674679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:47.674648    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"
Nov 04 00:56:47.674679 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:47.674672    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc" event={"ID":"03a02bca-ee13-48ed-90bd-a0b2e19373e5","Type":"ContainerDied","Data":"d7d4b5856024a993ea7d055ea671a650979cd7e5da92aace0d69716be018c6d9"}
Nov 04 00:56:47.674925 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:47.674715    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d4b5856024a993ea7d055ea671a650979cd7e5da92aace0d69716be018c6d9"
Nov 04 00:56:55.101211 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:55.101175    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"]
Nov 04 00:56:55.103181 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:55.103147    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b34223-mg8zc"]
Nov 04 00:56:57.016940 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:56:57.016904    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a02bca-ee13-48ed-90bd-a0b2e19373e5" path="/var/lib/kubelet/pods/03a02bca-ee13-48ed-90bd-a0b2e19373e5/volumes"
Nov 04 00:56:58.023105 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:56:58.023060    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:57:12.443534 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.443498    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"]
Nov 04 00:57:12.444051 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.443692    5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8" podUID="9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" containerName="manager" containerID="cri-o://f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82" gracePeriod=10
Nov 04 00:57:12.712511 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.712482    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 04 00:57:12.816655 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.816617    5074 generic.go:356] "Generic (PLEG): container finished" podID="9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" containerID="f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82" exitCode=0
Nov 04 00:57:12.816885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.816666    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"
Nov 04 00:57:12.816885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.816677    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8" event={"ID":"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14","Type":"ContainerDied","Data":"f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82"}
Nov 04 00:57:12.816885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.816765    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8" event={"ID":"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14","Type":"ContainerDied","Data":"188a1e80039604add7a422bafadc6261b7547b6fdeb09111f9a52696d95a24f0"}
Nov 04 00:57:12.816885 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.816789    5074 scope.go:122] "RemoveContainer" containerID="f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82"
Nov 04 00:57:12.830567 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.830546    5074 scope.go:122] "RemoveContainer" containerID="f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82"
Nov 04 00:57:12.830862 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:57:12.830831    5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82\": container with ID starting with f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82 not found: ID does not exist" containerID="f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82"
Nov 04 00:57:12.830916 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.830872    5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82"} err="failed to get container status \"f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82\": rpc error: code = NotFound desc = could not find container \"f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82\": container with ID starting with f54db8420e29cdeae593ddeeb9c99401338c57a8c02fbd9d69adef9a94f93e82 not found: ID does not exist"
Nov 04 00:57:12.842239 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.842219    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9\" (UniqueName: \"kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9\") pod \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") "
Nov 04 00:57:12.842358 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.842344    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume\") pod \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\" (UID: \"9e5dacd3-94a5-4e52-bdb4-26f2fe871a14\") "
Nov 04 00:57:12.842669 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.842646    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume" pod "9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" (UID: "9e5dacd3-94a5-4e52-bdb4-26f2fe871a14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:57:12.844302 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.844280    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9" pod "9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" (UID: "9e5dacd3-94a5-4e52-bdb4-26f2fe871a14"). InnerVolumeSpecName "kube-api-access-zbbg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:57:12.943974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.943936    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbg9\" (UniqueName: \"kubernetes.io/projected/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-kube-api-access-zbbg9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:57:12.943974 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:12.943969    5074 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14-extensions-socket-volume\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:57:13.137236 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:13.137204    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"]
Nov 04 00:57:13.141210 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:13.141180    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w2cn8"]
Nov 04 00:57:15.016058 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:57:15.016022    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" path="/var/lib/kubelet/pods/9e5dacd3-94a5-4e52-bdb4-26f2fe871a14/volumes"
Nov 04 00:57:28.021048 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:57:28.021010    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:57:58.020055 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:57:58.020011    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:58:13.502684 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.502646    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"]
Nov 04 00:58:13.503132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.503106    5074 cpu_manager.go:414] "containerMap: removing container" podUID="03a02bca-ee13-48ed-90bd-a0b2e19373e5" containerName="revoke-keys"
Nov 04 00:58:13.503132 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.503119    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a02bca-ee13-48ed-90bd-a0b2e19373e5" containerName="revoke-keys"
Nov 04 00:58:13.503212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.503139    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" containerName="manager"
Nov 04 00:58:13.503212 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.503162    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" containerName="manager"
Nov 04 00:58:13.503286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.503225    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5dacd3-94a5-4e52-bdb4-26f2fe871a14" containerName="manager"
Nov 04 00:58:13.503286 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.503239    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a02bca-ee13-48ed-90bd-a0b2e19373e5" containerName="revoke-keys"
Nov 04 00:58:13.506282 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.506258    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.514583 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.514556    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n88b2\""
Nov 04 00:58:13.517981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.517955    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"]
Nov 04 00:58:13.610556 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.610525    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5hg\" (UniqueName: \"kubernetes.io/projected/91e01291-c87c-47db-a09d-400263ff937c-kube-api-access-hr5hg\") pod \"kuadrant-operator-controller-manager-678d87766f-6pmdm\" (UID: \"91e01291-c87c-47db-a09d-400263ff937c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.610771 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.610595    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91e01291-c87c-47db-a09d-400263ff937c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6pmdm\" (UID: \"91e01291-c87c-47db-a09d-400263ff937c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.711780 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.711726    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5hg\" (UniqueName: \"kubernetes.io/projected/91e01291-c87c-47db-a09d-400263ff937c-kube-api-access-hr5hg\") pod \"kuadrant-operator-controller-manager-678d87766f-6pmdm\" (UID: \"91e01291-c87c-47db-a09d-400263ff937c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.711928 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.711812    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91e01291-c87c-47db-a09d-400263ff937c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6pmdm\" (UID: \"91e01291-c87c-47db-a09d-400263ff937c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.712265 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.712239    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91e01291-c87c-47db-a09d-400263ff937c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6pmdm\" (UID: \"91e01291-c87c-47db-a09d-400263ff937c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.727325 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.727291    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5hg\" (UniqueName: \"kubernetes.io/projected/91e01291-c87c-47db-a09d-400263ff937c-kube-api-access-hr5hg\") pod \"kuadrant-operator-controller-manager-678d87766f-6pmdm\" (UID: \"91e01291-c87c-47db-a09d-400263ff937c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:13.820452 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:13.820374    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:14.101581 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:14.101554    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"]
Nov 04 00:58:14.103780 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:58:14.103734    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e01291_c87c_47db_a09d_400263ff937c.slice/crio-59e467be6213b4ba6c159072f8fa8d5e4c66ed32c8c687eebbb8b5e3eb29b51e WatchSource:0}: Error finding container 59e467be6213b4ba6c159072f8fa8d5e4c66ed32c8c687eebbb8b5e3eb29b51e: Status 404 returned error can't find the container with id 59e467be6213b4ba6c159072f8fa8d5e4c66ed32c8c687eebbb8b5e3eb29b51e
Nov 04 00:58:14.121311 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:14.121285    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm" event={"ID":"91e01291-c87c-47db-a09d-400263ff937c","Type":"ContainerStarted","Data":"59e467be6213b4ba6c159072f8fa8d5e4c66ed32c8c687eebbb8b5e3eb29b51e"}
Nov 04 00:58:15.128068 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:15.128027    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm" event={"ID":"91e01291-c87c-47db-a09d-400263ff937c","Type":"ContainerStarted","Data":"f90c8b5ecd7868e4e7ffbf7081a4646d7d5fc8f7cc22d89d6edc3a2400d607d3"}
Nov 04 00:58:15.128498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:15.128142    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:15.155850 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:15.155799    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm" podStartSLOduration=2.1557873499999998 podStartE2EDuration="2.15578735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:15.153063041 +0000 UTC m=+5360.252663184" watchObservedRunningTime="2026-11-04 00:58:15.15578735 +0000 UTC m=+5360.255387494"
Nov 04 00:58:26.135091 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:26.135060    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pmdm"
Nov 04 00:58:28.022931 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:58:28.022892    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:58:34.065649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:34.064860    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:58:34.065649 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:34.064945    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:58:56.293513 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:56.293477    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:58:56.320830 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:58:56.320790    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 00:58:58.031500 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:58:58.031445    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:59:28.087770 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:59:28.087713    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 00:59:31.224151 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.224113    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"]
Nov 04 00:59:31.228000 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.227977    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.233920 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.233896    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:59:31.235694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.235410    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:59:31.238023 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.237997    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"]
Nov 04 00:59:31.353340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.353308    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lgj\" (UniqueName: \"kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.353523 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.353365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.353632 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.353610    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.454423 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.454378    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.454612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.454429    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lgj\" (UniqueName: \"kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.454612 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.454487    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.455207 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.455178    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.456768 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.456732    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.467187 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.467161    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lgj\" (UniqueName: \"kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj\") pod \"maas-api-revoke-sub-a237795a-8ksnf\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.543097 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.543033    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:31.816574 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:31.816550    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"]
Nov 04 00:59:31.819065 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:59:31.819036    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b198ee_1a28_4c23_92c6_11344e0f71db.slice/crio-f9694b06e31173c8472f534474893e78503072d7728e8ced9bbd5a1e09d8e09d WatchSource:0}: Error finding container f9694b06e31173c8472f534474893e78503072d7728e8ced9bbd5a1e09d8e09d: Status 404 returned error can't find the container with id f9694b06e31173c8472f534474893e78503072d7728e8ced9bbd5a1e09d8e09d
Nov 04 00:59:32.635292 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:32.635257    5074 generic.go:356] "Generic (PLEG): container finished" podID="43b198ee-1a28-4c23-92c6-11344e0f71db" containerID="fef697f24d645656d84b7717773c2631183c9e52f02fede8cd07ee6a2b6f725b" exitCode=0
Nov 04 00:59:32.635694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:32.635335    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf" event={"ID":"43b198ee-1a28-4c23-92c6-11344e0f71db","Type":"ContainerDied","Data":"fef697f24d645656d84b7717773c2631183c9e52f02fede8cd07ee6a2b6f725b"}
Nov 04 00:59:32.635694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:32.635398    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf" event={"ID":"43b198ee-1a28-4c23-92c6-11344e0f71db","Type":"ContainerStarted","Data":"f9694b06e31173c8472f534474893e78503072d7728e8ced9bbd5a1e09d8e09d"}
Nov 04 00:59:33.782672 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.782646    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:33.878694 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.878653    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca\") pod \"43b198ee-1a28-4c23-92c6-11344e0f71db\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") "
Nov 04 00:59:33.878915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.878717    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj\" (UniqueName: \"kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj\") pod \"43b198ee-1a28-4c23-92c6-11344e0f71db\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") "
Nov 04 00:59:33.878915 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.878899    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca\") pod \"43b198ee-1a28-4c23-92c6-11344e0f71db\" (UID: \"43b198ee-1a28-4c23-92c6-11344e0f71db\") "
Nov 04 00:59:33.879117 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.879093    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca" pod "43b198ee-1a28-4c23-92c6-11344e0f71db" (UID: "43b198ee-1a28-4c23-92c6-11344e0f71db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:59:33.879281 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.879254    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:33.881003 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.880966    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca" pod "43b198ee-1a28-4c23-92c6-11344e0f71db" (UID: "43b198ee-1a28-4c23-92c6-11344e0f71db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:59:33.881114 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.881068    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj" pod "43b198ee-1a28-4c23-92c6-11344e0f71db" (UID: "43b198ee-1a28-4c23-92c6-11344e0f71db"). InnerVolumeSpecName "kube-api-access-h9lgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:59:33.980283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.980244    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b198ee-1a28-4c23-92c6-11344e0f71db-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:33.980283 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:33.980280    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9lgj\" (UniqueName: \"kubernetes.io/projected/43b198ee-1a28-4c23-92c6-11344e0f71db-kube-api-access-h9lgj\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:34.647331 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:34.647305    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"
Nov 04 00:59:34.647498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:34.647334    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf" event={"ID":"43b198ee-1a28-4c23-92c6-11344e0f71db","Type":"ContainerDied","Data":"f9694b06e31173c8472f534474893e78503072d7728e8ced9bbd5a1e09d8e09d"}
Nov 04 00:59:34.647498 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:34.647385    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9694b06e31173c8472f534474893e78503072d7728e8ced9bbd5a1e09d8e09d"
Nov 04 00:59:41.042705 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.042610    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"]
Nov 04 00:59:41.043382 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.043361    5074 cpu_manager.go:414] "containerMap: removing container" podUID="43b198ee-1a28-4c23-92c6-11344e0f71db" containerName="revoke-keys"
Nov 04 00:59:41.043439 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.043385    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b198ee-1a28-4c23-92c6-11344e0f71db" containerName="revoke-keys"
Nov 04 00:59:41.043555 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.043540    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b198ee-1a28-4c23-92c6-11344e0f71db" containerName="revoke-keys"
Nov 04 00:59:41.047191 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.047134    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.052981 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.052958    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:59:41.053327 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.053268    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:59:41.054543 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.054516    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"]
Nov 04 00:59:41.155076 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.155043    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.155258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.155086    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tf5l\" (UniqueName: \"kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.155258 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.155156    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.256588 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.256549    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.256814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.256597    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tf5l\" (UniqueName: \"kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.256814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.256735    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.257559 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.257532    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.259340 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.259317    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.270713 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.270685    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tf5l\" (UniqueName: \"kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l\") pod \"maas-api-revoke-sub-8f4a30ad-zct5j\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.367697 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.367620    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:41.651610 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.651579    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"]
Nov 04 00:59:41.653505 ip-10-0-138-239 kubenswrapper[5074]: W1104 00:59:41.653473    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c12640_f7c7_4371_9f20_7c51a9d522e2.slice/crio-a5fbe989cdb1a260802bab89c470ab47a29ae26d1707c0d89c6ce63866418e9e WatchSource:0}: Error finding container a5fbe989cdb1a260802bab89c470ab47a29ae26d1707c0d89c6ce63866418e9e: Status 404 returned error can't find the container with id a5fbe989cdb1a260802bab89c470ab47a29ae26d1707c0d89c6ce63866418e9e
Nov 04 00:59:41.684763 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:41.684715    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j" event={"ID":"84c12640-f7c7-4371-9f20-7c51a9d522e2","Type":"ContainerStarted","Data":"a5fbe989cdb1a260802bab89c470ab47a29ae26d1707c0d89c6ce63866418e9e"}
Nov 04 00:59:42.691709 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:42.691671    5074 generic.go:356] "Generic (PLEG): container finished" podID="84c12640-f7c7-4371-9f20-7c51a9d522e2" containerID="d34276bbbba95aca9a878c6566fcdef74f4099389cac20ecb1ee0f8338192e91" exitCode=0
Nov 04 00:59:42.692162 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:42.691784    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j" event={"ID":"84c12640-f7c7-4371-9f20-7c51a9d522e2","Type":"ContainerDied","Data":"d34276bbbba95aca9a878c6566fcdef74f4099389cac20ecb1ee0f8338192e91"}
Nov 04 00:59:43.838568 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.838539    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:43.887420 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.887385    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca\") pod \"84c12640-f7c7-4371-9f20-7c51a9d522e2\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") "
Nov 04 00:59:43.887605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.887437    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca\") pod \"84c12640-f7c7-4371-9f20-7c51a9d522e2\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") "
Nov 04 00:59:43.887605 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.887461    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l\" (UniqueName: \"kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l\") pod \"84c12640-f7c7-4371-9f20-7c51a9d522e2\" (UID: \"84c12640-f7c7-4371-9f20-7c51a9d522e2\") "
Nov 04 00:59:43.887875 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.887852    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca" pod "84c12640-f7c7-4371-9f20-7c51a9d522e2" (UID: "84c12640-f7c7-4371-9f20-7c51a9d522e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:59:43.889675 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.889644    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca" pod "84c12640-f7c7-4371-9f20-7c51a9d522e2" (UID: "84c12640-f7c7-4371-9f20-7c51a9d522e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:59:43.889814 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.889717    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l" pod "84c12640-f7c7-4371-9f20-7c51a9d522e2" (UID: "84c12640-f7c7-4371-9f20-7c51a9d522e2"). InnerVolumeSpecName "kube-api-access-6tf5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:59:43.988967 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.988881    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:43.988967 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.988916    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tf5l\" (UniqueName: \"kubernetes.io/projected/84c12640-f7c7-4371-9f20-7c51a9d522e2-kube-api-access-6tf5l\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:43.988967 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:43.988927    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c12640-f7c7-4371-9f20-7c51a9d522e2-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:44.703189 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:44.703159    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"
Nov 04 00:59:44.703386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:44.703240    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j" event={"ID":"84c12640-f7c7-4371-9f20-7c51a9d522e2","Type":"ContainerDied","Data":"a5fbe989cdb1a260802bab89c470ab47a29ae26d1707c0d89c6ce63866418e9e"}
Nov 04 00:59:44.703386 ip-10-0-138-239 kubenswrapper[5074]: I1104 00:59:44.703288    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5fbe989cdb1a260802bab89c470ab47a29ae26d1707c0d89c6ce63866418e9e"
Nov 04 00:59:58.021190 ip-10-0-138-239 kubenswrapper[5074]: E1104 00:59:58.021152    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:00:00.135340 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.135303    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"]
Nov 04 01:00:00.135774 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.135764    5074 cpu_manager.go:414] "containerMap: removing container" podUID="84c12640-f7c7-4371-9f20-7c51a9d522e2" containerName="revoke-keys"
Nov 04 01:00:00.135839 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.135779    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c12640-f7c7-4371-9f20-7c51a9d522e2" containerName="revoke-keys"
Nov 04 01:00:00.135876 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.135866    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c12640-f7c7-4371-9f20-7c51a9d522e2" containerName="revoke-keys"
Nov 04 01:00:00.139587 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.139568    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.147362 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.147332    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"]
Nov 04 01:00:00.153491 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.153467    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:00:00.245572 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.245537    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895900-p2ktp\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.245773 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.245575    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl275\" (UniqueName: \"kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275\") pod \"maas-api-key-cleanup-29895900-p2ktp\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.346992 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.346959    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895900-p2ktp\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.346992 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.346992    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl275\" (UniqueName: \"kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275\") pod \"maas-api-key-cleanup-29895900-p2ktp\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.347623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.347600    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895900-p2ktp\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.360786 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.360755    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl275\" (UniqueName: \"kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275\") pod \"maas-api-key-cleanup-29895900-p2ktp\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.457151 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.457118    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:00.734152 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.734120    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"]
Nov 04 01:00:00.736348 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:00:00.736314    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70077cbb_c084_4995_9450_fb5f4947ffc2.slice/crio-200236c6db61cd403a839115629a2557e18e121c0fc2db3163ff4e776f3440b8 WatchSource:0}: Error finding container 200236c6db61cd403a839115629a2557e18e121c0fc2db3163ff4e776f3440b8: Status 404 returned error can't find the container with id 200236c6db61cd403a839115629a2557e18e121c0fc2db3163ff4e776f3440b8
Nov 04 01:00:00.785357 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:00.785325    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" event={"ID":"70077cbb-c084-4995-9450-fb5f4947ffc2","Type":"ContainerStarted","Data":"200236c6db61cd403a839115629a2557e18e121c0fc2db3163ff4e776f3440b8"}
Nov 04 01:00:01.791590 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:01.791495    5074 generic.go:356] "Generic (PLEG): container finished" podID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerID="99825f0e26a38f19773ea9b0555ab4a87fd8f636fb76a3de05cef147e3c3bfeb" exitCode=60
Nov 04 01:00:01.791590 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:01.791579    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" event={"ID":"70077cbb-c084-4995-9450-fb5f4947ffc2","Type":"ContainerDied","Data":"99825f0e26a38f19773ea9b0555ab4a87fd8f636fb76a3de05cef147e3c3bfeb"}
Nov 04 01:00:01.792147 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:01.791907    5074 scope.go:122] "RemoveContainer" containerID="99825f0e26a38f19773ea9b0555ab4a87fd8f636fb76a3de05cef147e3c3bfeb"
Nov 04 01:00:02.798721 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:02.798641    5074 generic.go:356] "Generic (PLEG): container finished" podID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerID="7343bed21039c8b97ab931c946655266572783e2fdd49c150b9ba3e8ad616a60" exitCode=60
Nov 04 01:00:02.799202 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:02.798729    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" event={"ID":"70077cbb-c084-4995-9450-fb5f4947ffc2","Type":"ContainerDied","Data":"7343bed21039c8b97ab931c946655266572783e2fdd49c150b9ba3e8ad616a60"}
Nov 04 01:00:02.799202 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:02.798817    5074 scope.go:122] "RemoveContainer" containerID="99825f0e26a38f19773ea9b0555ab4a87fd8f636fb76a3de05cef147e3c3bfeb"
Nov 04 01:00:02.799202 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:02.799080    5074 scope.go:122] "RemoveContainer" containerID="7343bed21039c8b97ab931c946655266572783e2fdd49c150b9ba3e8ad616a60"
Nov 04 01:00:02.799359 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:00:02.799333    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895900-p2ktp_odh-ai-gateway-infra(70077cbb-c084-4995-9450-fb5f4947ffc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2"
Nov 04 01:00:03.806561 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:03.806534    5074 scope.go:122] "RemoveContainer" containerID="7343bed21039c8b97ab931c946655266572783e2fdd49c150b9ba3e8ad616a60"
Nov 04 01:00:03.807008 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:00:03.806797    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895900-p2ktp_odh-ai-gateway-infra(70077cbb-c084-4995-9450-fb5f4947ffc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2"
Nov 04 01:00:14.009884 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:14.009853    5074 scope.go:122] "RemoveContainer" containerID="7343bed21039c8b97ab931c946655266572783e2fdd49c150b9ba3e8ad616a60"
Nov 04 01:00:14.908383 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:14.908347    5074 generic.go:356] "Generic (PLEG): container finished" podID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerID="89f3e6d69e34d44b510da48edbc98cf3c30f51d758da09275de08cc8eb40dd1a" exitCode=60
Nov 04 01:00:14.908516 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:14.908419    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" event={"ID":"70077cbb-c084-4995-9450-fb5f4947ffc2","Type":"ContainerDied","Data":"89f3e6d69e34d44b510da48edbc98cf3c30f51d758da09275de08cc8eb40dd1a"}
Nov 04 01:00:14.908516 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:14.908478    5074 scope.go:122] "RemoveContainer" containerID="7343bed21039c8b97ab931c946655266572783e2fdd49c150b9ba3e8ad616a60"
Nov 04 01:00:14.908879 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:14.908856    5074 scope.go:122] "RemoveContainer" containerID="89f3e6d69e34d44b510da48edbc98cf3c30f51d758da09275de08cc8eb40dd1a"
Nov 04 01:00:14.909157 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:00:14.909137    5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895900-p2ktp_odh-ai-gateway-infra(70077cbb-c084-4995-9450-fb5f4947ffc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2"
Nov 04 01:00:15.939137 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:15.939104    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"]
Nov 04 01:00:16.085311 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.085284    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:16.240062 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.239985    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca\") pod \"70077cbb-c084-4995-9450-fb5f4947ffc2\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") "
Nov 04 01:00:16.240062 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.240029    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275\" (UniqueName: \"kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275\") pod \"70077cbb-c084-4995-9450-fb5f4947ffc2\" (UID: \"70077cbb-c084-4995-9450-fb5f4947ffc2\") "
Nov 04 01:00:16.240372 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.240347    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca" pod "70077cbb-c084-4995-9450-fb5f4947ffc2" (UID: "70077cbb-c084-4995-9450-fb5f4947ffc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:00:16.242124 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.242097    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275" pod "70077cbb-c084-4995-9450-fb5f4947ffc2" (UID: "70077cbb-c084-4995-9450-fb5f4947ffc2"). InnerVolumeSpecName "kube-api-access-jl275". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:00:16.341083 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.341052    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70077cbb-c084-4995-9450-fb5f4947ffc2-maas-api-service-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:00:16.341083 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.341080    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl275\" (UniqueName: \"kubernetes.io/projected/70077cbb-c084-4995-9450-fb5f4947ffc2-kube-api-access-jl275\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:00:16.922164 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.922136    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"
Nov 04 01:00:16.922356 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.922134    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp" event={"ID":"70077cbb-c084-4995-9450-fb5f4947ffc2","Type":"ContainerDied","Data":"200236c6db61cd403a839115629a2557e18e121c0fc2db3163ff4e776f3440b8"}
Nov 04 01:00:16.922356 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.922313    5074 scope.go:122] "RemoveContainer" containerID="89f3e6d69e34d44b510da48edbc98cf3c30f51d758da09275de08cc8eb40dd1a"
Nov 04 01:00:16.946544 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.946520    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"]
Nov 04 01:00:16.951852 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:16.951830    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895900-p2ktp"]
Nov 04 01:00:17.016353 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:17.016329    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" path="/var/lib/kubelet/pods/70077cbb-c084-4995-9450-fb5f4947ffc2/volumes"
Nov 04 01:00:28.020053 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:00:28.020014    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:00:48.027021 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.026984    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"]
Nov 04 01:00:48.029250 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.029226    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b45680a-m78vt"]
Nov 04 01:00:48.601042 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601003    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"]
Nov 04 01:00:48.601502 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601485    5074 cpu_manager.go:414] "containerMap: removing container" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601553 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601503    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601553 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601520    5074 cpu_manager.go:414] "containerMap: removing container" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601553 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601525    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601553 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601536    5074 cpu_manager.go:414] "containerMap: removing container" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601553 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601541    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601767 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601619    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.601767 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.601629    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:48.604323 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.604305    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.610538 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.610512    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:00:48.613465 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.613443    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:00:48.615733 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.615702    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"]
Nov 04 01:00:48.670985 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.670958    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.671110 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.671018    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.671110 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.671094    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtxt\" (UniqueName: \"kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.771812 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.771780    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.771983 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.771826    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtxt\" (UniqueName: \"kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.771983 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.771907    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.772545 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.772519    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.774128 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.774099    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.785200 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.785175    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtxt\" (UniqueName: \"kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt\") pod \"maas-api-revoke-sub-eac89823-m9wg5\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:48.921617 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:48.921531    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:49.017242 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:49.017205    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22842901-61fb-47ff-9d3e-96eae1d7bbb5" path="/var/lib/kubelet/pods/22842901-61fb-47ff-9d3e-96eae1d7bbb5/volumes"
Nov 04 01:00:49.200308 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:49.200281    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"]
Nov 04 01:00:49.202855 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:00:49.202812    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b626d5a_6d3e_4edf_9448_9e6e77bf7784.slice/crio-01c39e9e009677d5006e81d91d4cd3cc61ecd2b6e4af0714c6fe50eef1e70dac WatchSource:0}: Error finding container 01c39e9e009677d5006e81d91d4cd3cc61ecd2b6e4af0714c6fe50eef1e70dac: Status 404 returned error can't find the container with id 01c39e9e009677d5006e81d91d4cd3cc61ecd2b6e4af0714c6fe50eef1e70dac
Nov 04 01:00:50.129510 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:50.129473    5074 generic.go:356] "Generic (PLEG): container finished" podID="5b626d5a-6d3e-4edf-9448-9e6e77bf7784" containerID="097776caa62412e54a330772aa345f0635d729855e4cdc5504107d9c673c52d6" exitCode=0
Nov 04 01:00:50.129699 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:50.129560    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5" event={"ID":"5b626d5a-6d3e-4edf-9448-9e6e77bf7784","Type":"ContainerDied","Data":"097776caa62412e54a330772aa345f0635d729855e4cdc5504107d9c673c52d6"}
Nov 04 01:00:50.129699 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:50.129618    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5" event={"ID":"5b626d5a-6d3e-4edf-9448-9e6e77bf7784","Type":"ContainerStarted","Data":"01c39e9e009677d5006e81d91d4cd3cc61ecd2b6e4af0714c6fe50eef1e70dac"}
Nov 04 01:00:51.280550 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.280522    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:51.398608 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.398506    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt\" (UniqueName: \"kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt\") pod \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") "
Nov 04 01:00:51.398608 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.398607    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca\") pod \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") "
Nov 04 01:00:51.398925 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.398629    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca\") pod \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\" (UID: \"5b626d5a-6d3e-4edf-9448-9e6e77bf7784\") "
Nov 04 01:00:51.399069 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.399045    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca" pod "5b626d5a-6d3e-4edf-9448-9e6e77bf7784" (UID: "5b626d5a-6d3e-4edf-9448-9e6e77bf7784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:00:51.400768 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.400728    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca" pod "5b626d5a-6d3e-4edf-9448-9e6e77bf7784" (UID: "5b626d5a-6d3e-4edf-9448-9e6e77bf7784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:00:51.400874 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.400775    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt" pod "5b626d5a-6d3e-4edf-9448-9e6e77bf7784" (UID: "5b626d5a-6d3e-4edf-9448-9e6e77bf7784"). InnerVolumeSpecName "kube-api-access-ggtxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:00:51.500181 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.500144    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:00:51.500181 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.500177    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:00:51.500181 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:51.500187    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtxt\" (UniqueName: \"kubernetes.io/projected/5b626d5a-6d3e-4edf-9448-9e6e77bf7784-kube-api-access-ggtxt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:00:52.143790 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:52.143760    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"
Nov 04 01:00:52.143790 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:52.143770    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5" event={"ID":"5b626d5a-6d3e-4edf-9448-9e6e77bf7784","Type":"ContainerDied","Data":"01c39e9e009677d5006e81d91d4cd3cc61ecd2b6e4af0714c6fe50eef1e70dac"}
Nov 04 01:00:52.144055 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:52.143825    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c39e9e009677d5006e81d91d4cd3cc61ecd2b6e4af0714c6fe50eef1e70dac"
Nov 04 01:00:58.020118 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:00:58.020080    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:00:58.833208 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.833167    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"]
Nov 04 01:00:58.833937 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.833910    5074 cpu_manager.go:414] "containerMap: removing container" podUID="5b626d5a-6d3e-4edf-9448-9e6e77bf7784" containerName="revoke-keys"
Nov 04 01:00:58.833937 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.833936    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b626d5a-6d3e-4edf-9448-9e6e77bf7784" containerName="revoke-keys"
Nov 04 01:00:58.834154 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.834054    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="70077cbb-c084-4995-9450-fb5f4947ffc2" containerName="cleanup"
Nov 04 01:00:58.834154 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.834066    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b626d5a-6d3e-4edf-9448-9e6e77bf7784" containerName="revoke-keys"
Nov 04 01:00:58.837995 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.837973    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:58.843732 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.843709    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:00:58.843732 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.843910    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:00:58.846341 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.846310    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"]
Nov 04 01:00:58.970939 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.970895    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpv75\" (UniqueName: \"kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:58.971171 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.971037    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:58.971171 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:58.971081    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.072015 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.071963    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.072500 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.072045    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.072500 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.072128    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpv75\" (UniqueName: \"kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.073012 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.072984    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.074472 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.074446    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.084935 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.084879    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpv75\" (UniqueName: \"kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75\") pod \"maas-api-revoke-sub-36e9ad82-p2ngq\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.154221 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.154182    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:00:59.438910 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:00:59.438882    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"]
Nov 04 01:00:59.441561 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:00:59.441525    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038a62eb_a9e0_426d_aee0_6a161bcf619c.slice/crio-38b18c7eb02a6c50adfec91462f48b06e7ed48079b73b68a608a6ced07b95f55 WatchSource:0}: Error finding container 38b18c7eb02a6c50adfec91462f48b06e7ed48079b73b68a608a6ced07b95f55: Status 404 returned error can't find the container with id 38b18c7eb02a6c50adfec91462f48b06e7ed48079b73b68a608a6ced07b95f55
Nov 04 01:01:00.186337 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:00.186303    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq" event={"ID":"038a62eb-a9e0-426d-aee0-6a161bcf619c","Type":"ContainerStarted","Data":"9a69eca20e1e01fb25608e0fe820da98e5bf3fc9ca9b0b138beb4326a82cf55f"}
Nov 04 01:01:00.186727 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:00.186368    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq" event={"ID":"038a62eb-a9e0-426d-aee0-6a161bcf619c","Type":"ContainerStarted","Data":"38b18c7eb02a6c50adfec91462f48b06e7ed48079b73b68a608a6ced07b95f55"}
Nov 04 01:01:00.205784 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:00.205711    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq" podStartSLOduration=2.2056973810000002 podStartE2EDuration="2.205697381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:01:00.204984895 +0000 UTC m=+5525.304585052" watchObservedRunningTime="2026-11-04 01:01:00.205697381 +0000 UTC m=+5525.305297524"
Nov 04 01:01:01.193369 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:01.193335    5074 generic.go:356] "Generic (PLEG): container finished" podID="038a62eb-a9e0-426d-aee0-6a161bcf619c" containerID="9a69eca20e1e01fb25608e0fe820da98e5bf3fc9ca9b0b138beb4326a82cf55f" exitCode=0
Nov 04 01:01:01.193863 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:01.193418    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq" event={"ID":"038a62eb-a9e0-426d-aee0-6a161bcf619c","Type":"ContainerDied","Data":"9a69eca20e1e01fb25608e0fe820da98e5bf3fc9ca9b0b138beb4326a82cf55f"}
Nov 04 01:01:02.351807 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.351778    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:01:02.504025 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.503912    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75\" (UniqueName: \"kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75\") pod \"038a62eb-a9e0-426d-aee0-6a161bcf619c\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") "
Nov 04 01:01:02.504025 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.503957    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca\") pod \"038a62eb-a9e0-426d-aee0-6a161bcf619c\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") "
Nov 04 01:01:02.504310 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.504093    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca\") pod \"038a62eb-a9e0-426d-aee0-6a161bcf619c\" (UID: \"038a62eb-a9e0-426d-aee0-6a161bcf619c\") "
Nov 04 01:01:02.505149 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.505120    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca" pod "038a62eb-a9e0-426d-aee0-6a161bcf619c" (UID: "038a62eb-a9e0-426d-aee0-6a161bcf619c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:01:02.507358 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.507333    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75" pod "038a62eb-a9e0-426d-aee0-6a161bcf619c" (UID: "038a62eb-a9e0-426d-aee0-6a161bcf619c"). InnerVolumeSpecName "kube-api-access-wpv75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:01:02.507451 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.507367    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca" pod "038a62eb-a9e0-426d-aee0-6a161bcf619c" (UID: "038a62eb-a9e0-426d-aee0-6a161bcf619c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:01:02.604984 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.604928    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpv75\" (UniqueName: \"kubernetes.io/projected/038a62eb-a9e0-426d-aee0-6a161bcf619c-kube-api-access-wpv75\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:02.604984 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.604980    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:02.604984 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:02.604992    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038a62eb-a9e0-426d-aee0-6a161bcf619c-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:03.205086 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:03.205056    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"
Nov 04 01:01:03.205264 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:03.205100    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq" event={"ID":"038a62eb-a9e0-426d-aee0-6a161bcf619c","Type":"ContainerDied","Data":"38b18c7eb02a6c50adfec91462f48b06e7ed48079b73b68a608a6ced07b95f55"}
Nov 04 01:01:03.205264 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:03.205144    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38b18c7eb02a6c50adfec91462f48b06e7ed48079b73b68a608a6ced07b95f55"
Nov 04 01:01:26.852151 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.852073    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"]
Nov 04 01:01:26.855031 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.854994    5074 cpu_manager.go:414] "containerMap: removing container" podUID="038a62eb-a9e0-426d-aee0-6a161bcf619c" containerName="revoke-keys"
Nov 04 01:01:26.855031 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.855029    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038a62eb-a9e0-426d-aee0-6a161bcf619c" containerName="revoke-keys"
Nov 04 01:01:26.857326 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.855258    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="038a62eb-a9e0-426d-aee0-6a161bcf619c" containerName="revoke-keys"
Nov 04 01:01:26.862386 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.862332    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:26.863035 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.862991    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"]
Nov 04 01:01:26.867659 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.867637    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:01:26.869005 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.868986    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:01:26.901231 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.901189    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7b9\" (UniqueName: \"kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:26.901420 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.901250    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:26.901420 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:26.901403    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.002817 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.002775    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7b9\" (UniqueName: \"kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.003020 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.002835    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.003020 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.002873    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.003732 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.003699    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.005320 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.005291    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.016826 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.016797    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7b9\" (UniqueName: \"kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9\") pod \"maas-api-revoke-sub-d1430007-rdtxf\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.182460 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.182365    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:27.469258 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:27.469230    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"]
Nov 04 01:01:27.471182 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:01:27.471139    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0516671c_96d7_4255_81e1_4df6e378ba09.slice/crio-8a30d23c547e179da231efad5948b3c92a264a1b1a7359f856a05c7940e7762b WatchSource:0}: Error finding container 8a30d23c547e179da231efad5948b3c92a264a1b1a7359f856a05c7940e7762b: Status 404 returned error can't find the container with id 8a30d23c547e179da231efad5948b3c92a264a1b1a7359f856a05c7940e7762b
Nov 04 01:01:28.022217 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:01:28.022179    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:01:28.408063 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:28.408026    5074 generic.go:356] "Generic (PLEG): container finished" podID="0516671c-96d7-4255-81e1-4df6e378ba09" containerID="76ba7e497c5dd3a7514807159fe78844a1ee137773d6c1138feeb46acc81d62a" exitCode=0
Nov 04 01:01:28.408286 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:28.408081    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf" event={"ID":"0516671c-96d7-4255-81e1-4df6e378ba09","Type":"ContainerDied","Data":"76ba7e497c5dd3a7514807159fe78844a1ee137773d6c1138feeb46acc81d62a"}
Nov 04 01:01:28.408286 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:28.408143    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf" event={"ID":"0516671c-96d7-4255-81e1-4df6e378ba09","Type":"ContainerStarted","Data":"8a30d23c547e179da231efad5948b3c92a264a1b1a7359f856a05c7940e7762b"}
Nov 04 01:01:29.562216 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.562191    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:29.629779 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.629729    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca\") pod \"0516671c-96d7-4255-81e1-4df6e378ba09\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") "
Nov 04 01:01:29.629940 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.629811    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca\") pod \"0516671c-96d7-4255-81e1-4df6e378ba09\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") "
Nov 04 01:01:29.629940 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.629907    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9\" (UniqueName: \"kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9\") pod \"0516671c-96d7-4255-81e1-4df6e378ba09\" (UID: \"0516671c-96d7-4255-81e1-4df6e378ba09\") "
Nov 04 01:01:29.630166 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.630139    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca" pod "0516671c-96d7-4255-81e1-4df6e378ba09" (UID: "0516671c-96d7-4255-81e1-4df6e378ba09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:01:29.631891 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.631861    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca" pod "0516671c-96d7-4255-81e1-4df6e378ba09" (UID: "0516671c-96d7-4255-81e1-4df6e378ba09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:01:29.631989 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.631937    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9" pod "0516671c-96d7-4255-81e1-4df6e378ba09" (UID: "0516671c-96d7-4255-81e1-4df6e378ba09"). InnerVolumeSpecName "kube-api-access-tq7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:01:29.730785 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.730732    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7b9\" (UniqueName: \"kubernetes.io/projected/0516671c-96d7-4255-81e1-4df6e378ba09-kube-api-access-tq7b9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:29.730785 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.730782    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:29.730991 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:29.730792    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0516671c-96d7-4255-81e1-4df6e378ba09-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:30.420180 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:30.420140    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf" event={"ID":"0516671c-96d7-4255-81e1-4df6e378ba09","Type":"ContainerDied","Data":"8a30d23c547e179da231efad5948b3c92a264a1b1a7359f856a05c7940e7762b"}
Nov 04 01:01:30.420180 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:30.420174    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"
Nov 04 01:01:30.420417 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:30.420199    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a30d23c547e179da231efad5948b3c92a264a1b1a7359f856a05c7940e7762b"
Nov 04 01:01:37.081084 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.081048    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"]
Nov 04 01:01:37.081582 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.081564    5074 cpu_manager.go:414] "containerMap: removing container" podUID="0516671c-96d7-4255-81e1-4df6e378ba09" containerName="revoke-keys"
Nov 04 01:01:37.081643 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.081583    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0516671c-96d7-4255-81e1-4df6e378ba09" containerName="revoke-keys"
Nov 04 01:01:37.081694 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.081687    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0516671c-96d7-4255-81e1-4df6e378ba09" containerName="revoke-keys"
Nov 04 01:01:37.085389 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.085360    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.095130 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.095100    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:01:37.095290 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.095201    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:01:37.095780 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.095705    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"]
Nov 04 01:01:37.200494 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.200455    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.200673 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.200513    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.200725 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.200667    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qc66\" (UniqueName: \"kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.301246 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.301205    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qc66\" (UniqueName: \"kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.301469 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.301300    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.301469 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.301353    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.302275 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.302246    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.303707 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.303671    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.314043 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.314017    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qc66\" (UniqueName: \"kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66\") pod \"maas-api-revoke-sub-8842b7b2-pzzg7\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.404660 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.404578    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:37.683961 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:37.683931    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"]
Nov 04 01:01:37.685977 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:01:37.685950    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8f8558_1e83_42ae_8538_ef7904dd7496.slice/crio-0183219d91167369664b8d0dbf06413330da755bcde80cde630dc421e782c3ff WatchSource:0}: Error finding container 0183219d91167369664b8d0dbf06413330da755bcde80cde630dc421e782c3ff: Status 404 returned error can't find the container with id 0183219d91167369664b8d0dbf06413330da755bcde80cde630dc421e782c3ff
Nov 04 01:01:38.471629 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:38.471576    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7" event={"ID":"be8f8558-1e83-42ae-8538-ef7904dd7496","Type":"ContainerStarted","Data":"2dd373318525119ff207a77d88f3109edb183a1bc500e83b3f0aed5d0c1c0d07"}
Nov 04 01:01:38.472183 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:38.471694    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7" event={"ID":"be8f8558-1e83-42ae-8538-ef7904dd7496","Type":"ContainerStarted","Data":"0183219d91167369664b8d0dbf06413330da755bcde80cde630dc421e782c3ff"}
Nov 04 01:01:38.493733 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:38.493671    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7" podStartSLOduration=1.493657035 podStartE2EDuration="1.493657035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:01:38.491327737 +0000 UTC m=+5563.590927883" watchObservedRunningTime="2026-11-04 01:01:38.493657035 +0000 UTC m=+5563.593257179"
Nov 04 01:01:39.494127 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:39.494084    5074 generic.go:356] "Generic (PLEG): container finished" podID="be8f8558-1e83-42ae-8538-ef7904dd7496" containerID="2dd373318525119ff207a77d88f3109edb183a1bc500e83b3f0aed5d0c1c0d07" exitCode=0
Nov 04 01:01:39.494669 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:39.494212    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7" event={"ID":"be8f8558-1e83-42ae-8538-ef7904dd7496","Type":"ContainerDied","Data":"2dd373318525119ff207a77d88f3109edb183a1bc500e83b3f0aed5d0c1c0d07"}
Nov 04 01:01:40.919728 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:40.919700    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:40.997102 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:40.997066    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca\") pod \"be8f8558-1e83-42ae-8538-ef7904dd7496\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") "
Nov 04 01:01:40.997273 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:40.997133    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca\") pod \"be8f8558-1e83-42ae-8538-ef7904dd7496\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") "
Nov 04 01:01:40.997534 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:40.997505    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca" pod "be8f8558-1e83-42ae-8538-ef7904dd7496" (UID: "be8f8558-1e83-42ae-8538-ef7904dd7496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:01:40.999292 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:40.999266    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca" pod "be8f8558-1e83-42ae-8538-ef7904dd7496" (UID: "be8f8558-1e83-42ae-8538-ef7904dd7496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:01:41.098071 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.098030    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66\" (UniqueName: \"kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66\") pod \"be8f8558-1e83-42ae-8538-ef7904dd7496\" (UID: \"be8f8558-1e83-42ae-8538-ef7904dd7496\") "
Nov 04 01:01:41.098304 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.098290    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:41.098356 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.098306    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8f8558-1e83-42ae-8538-ef7904dd7496-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:41.100363 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.100315    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66" pod "be8f8558-1e83-42ae-8538-ef7904dd7496" (UID: "be8f8558-1e83-42ae-8538-ef7904dd7496"). InnerVolumeSpecName "kube-api-access-8qc66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:01:41.199733 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.199676    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qc66\" (UniqueName: \"kubernetes.io/projected/be8f8558-1e83-42ae-8538-ef7904dd7496-kube-api-access-8qc66\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:01:41.516536 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.516456    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7" event={"ID":"be8f8558-1e83-42ae-8538-ef7904dd7496","Type":"ContainerDied","Data":"0183219d91167369664b8d0dbf06413330da755bcde80cde630dc421e782c3ff"}
Nov 04 01:01:41.516536 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.516504    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0183219d91167369664b8d0dbf06413330da755bcde80cde630dc421e782c3ff"
Nov 04 01:01:41.516536 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:01:41.516465    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"
Nov 04 01:01:58.020988 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:01:58.020941    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:02:02.569286 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:02.569249    5074 scope.go:122] "RemoveContainer" containerID="0b8f3e3bfb6b28ef34e8feae9933edb3465dcddbfbbf6a51a4ac745232b10dfd"
Nov 04 01:02:19.064764 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:19.059834    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:02:19.064764 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:19.059951    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:02:20.770298 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.770261    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"]
Nov 04 01:02:20.770701 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.770688    5074 cpu_manager.go:414] "containerMap: removing container" podUID="be8f8558-1e83-42ae-8538-ef7904dd7496" containerName="revoke-keys"
Nov 04 01:02:20.770701 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.770699    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8f8558-1e83-42ae-8538-ef7904dd7496" containerName="revoke-keys"
Nov 04 01:02:20.770816 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.770806    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8f8558-1e83-42ae-8538-ef7904dd7496" containerName="revoke-keys"
Nov 04 01:02:20.773680 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.773663    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:20.779527 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.779489    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:02:20.782197 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.782175    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:02:20.785421 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.785396    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"]
Nov 04 01:02:20.899467 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.899430    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:20.899697 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.899491    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cf9\" (UniqueName: \"kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:20.899697 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:20.899598    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.000642 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.000597    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.000873 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.000722    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.000873 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.000796    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cf9\" (UniqueName: \"kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.001412 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.001390    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.003035 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.003017    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.014055 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.014032    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cf9\" (UniqueName: \"kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9\") pod \"maas-api-revoke-sub-1efc0ce8-gtsvx\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.091446 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.091364    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:21.367513 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.367484    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"]
Nov 04 01:02:21.368915 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:02:21.368889    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4274aba_0c64_481b_be04_5462b6924c60.slice/crio-4c4aa59e0b97b09cd1430dd6ea7611119b6486b1c3304f727ffc017e2d0cf65a WatchSource:0}: Error finding container 4c4aa59e0b97b09cd1430dd6ea7611119b6486b1c3304f727ffc017e2d0cf65a: Status 404 returned error can't find the container with id 4c4aa59e0b97b09cd1430dd6ea7611119b6486b1c3304f727ffc017e2d0cf65a
Nov 04 01:02:21.805350 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.805316    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx" event={"ID":"b4274aba-0c64-481b-be04-5462b6924c60","Type":"ContainerStarted","Data":"73ee4cd3debca3df467f2d8ad1d009e519c17e6cfd1f98516f28b5b5ec6edffd"}
Nov 04 01:02:21.805761 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.805366    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx" event={"ID":"b4274aba-0c64-481b-be04-5462b6924c60","Type":"ContainerStarted","Data":"4c4aa59e0b97b09cd1430dd6ea7611119b6486b1c3304f727ffc017e2d0cf65a"}
Nov 04 01:02:21.826433 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:21.826382    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx" podStartSLOduration=1.8263691579999999 podStartE2EDuration="1.826369158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:21.824565926 +0000 UTC m=+5606.924166071" watchObservedRunningTime="2026-11-04 01:02:21.826369158 +0000 UTC m=+5606.925969301"
Nov 04 01:02:22.812213 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:22.812172    5074 generic.go:356] "Generic (PLEG): container finished" podID="b4274aba-0c64-481b-be04-5462b6924c60" containerID="73ee4cd3debca3df467f2d8ad1d009e519c17e6cfd1f98516f28b5b5ec6edffd" exitCode=0
Nov 04 01:02:22.812607 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:22.812256    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx" event={"ID":"b4274aba-0c64-481b-be04-5462b6924c60","Type":"ContainerDied","Data":"73ee4cd3debca3df467f2d8ad1d009e519c17e6cfd1f98516f28b5b5ec6edffd"}
Nov 04 01:02:23.953058 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:23.953030    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:24.028116 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.028085    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9\" (UniqueName: \"kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9\") pod \"b4274aba-0c64-481b-be04-5462b6924c60\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") "
Nov 04 01:02:24.028297 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.028184    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca\") pod \"b4274aba-0c64-481b-be04-5462b6924c60\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") "
Nov 04 01:02:24.028297 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.028245    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca\") pod \"b4274aba-0c64-481b-be04-5462b6924c60\" (UID: \"b4274aba-0c64-481b-be04-5462b6924c60\") "
Nov 04 01:02:24.028623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.028597    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca" pod "b4274aba-0c64-481b-be04-5462b6924c60" (UID: "b4274aba-0c64-481b-be04-5462b6924c60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:02:24.030184 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.030161    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca" pod "b4274aba-0c64-481b-be04-5462b6924c60" (UID: "b4274aba-0c64-481b-be04-5462b6924c60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:02:24.030287 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.030226    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9" pod "b4274aba-0c64-481b-be04-5462b6924c60" (UID: "b4274aba-0c64-481b-be04-5462b6924c60"). InnerVolumeSpecName "kube-api-access-n9cf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:02:24.129114 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.129031    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:24.129114 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.129061    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9cf9\" (UniqueName: \"kubernetes.io/projected/b4274aba-0c64-481b-be04-5462b6924c60-kube-api-access-n9cf9\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:24.129114 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.129070    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4274aba-0c64-481b-be04-5462b6924c60-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:24.823875 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.823848    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"
Nov 04 01:02:24.824061 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.823886    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx" event={"ID":"b4274aba-0c64-481b-be04-5462b6924c60","Type":"ContainerDied","Data":"4c4aa59e0b97b09cd1430dd6ea7611119b6486b1c3304f727ffc017e2d0cf65a"}
Nov 04 01:02:24.824061 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:24.823933    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4aa59e0b97b09cd1430dd6ea7611119b6486b1c3304f727ffc017e2d0cf65a"
Nov 04 01:02:28.021062 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:02:28.021023    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:02:31.049929 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.049894    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"]
Nov 04 01:02:31.050496 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.050378    5074 cpu_manager.go:414] "containerMap: removing container" podUID="b4274aba-0c64-481b-be04-5462b6924c60" containerName="revoke-keys"
Nov 04 01:02:31.050496 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.050392    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4274aba-0c64-481b-be04-5462b6924c60" containerName="revoke-keys"
Nov 04 01:02:31.050496 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.050477    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4274aba-0c64-481b-be04-5462b6924c60" containerName="revoke-keys"
Nov 04 01:02:31.053960 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.053938    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.060521 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.059406    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:02:31.062440 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.062408    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:02:31.065727 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.065705    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"]
Nov 04 01:02:31.197550 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.197510    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmw5\" (UniqueName: \"kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.197725 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.197567    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.197725 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.197650    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.299001 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.298955    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmw5\" (UniqueName: \"kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.299215 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.299018    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.299215 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.299053    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.299788 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.299726    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.301600 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.301543    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.312334 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.312313    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmw5\" (UniqueName: \"kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5\") pod \"maas-api-revoke-sub-3b5b3ff2-24k6m\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.370286 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.370253    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:31.649123 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.649088    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"]
Nov 04 01:02:31.651321 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:02:31.651275    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d80979_9d84_49d8_9f7b_efde82ebb7b2.slice/crio-05e491a3c2a2d9f0f46bb873574fd4d6f499d620b7e0924d47d7f49c13d7b830 WatchSource:0}: Error finding container 05e491a3c2a2d9f0f46bb873574fd4d6f499d620b7e0924d47d7f49c13d7b830: Status 404 returned error can't find the container with id 05e491a3c2a2d9f0f46bb873574fd4d6f499d620b7e0924d47d7f49c13d7b830
Nov 04 01:02:31.864577 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:31.864543    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m" event={"ID":"11d80979-9d84-49d8-9f7b-efde82ebb7b2","Type":"ContainerStarted","Data":"05e491a3c2a2d9f0f46bb873574fd4d6f499d620b7e0924d47d7f49c13d7b830"}
Nov 04 01:02:32.873458 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:32.873360    5074 generic.go:356] "Generic (PLEG): container finished" podID="11d80979-9d84-49d8-9f7b-efde82ebb7b2" containerID="dbefa091f29424037498f1026490288df98c2d2674551878dc4cca852412881f" exitCode=0
Nov 04 01:02:32.873458 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:32.873446    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m" event={"ID":"11d80979-9d84-49d8-9f7b-efde82ebb7b2","Type":"ContainerDied","Data":"dbefa091f29424037498f1026490288df98c2d2674551878dc4cca852412881f"}
Nov 04 01:02:34.503359 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.503035    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:34.616775 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.609103    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5\" (UniqueName: \"kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5\") pod \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") "
Nov 04 01:02:34.616775 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.609191    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca\") pod \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") "
Nov 04 01:02:34.616775 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.609238    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca\") pod \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\" (UID: \"11d80979-9d84-49d8-9f7b-efde82ebb7b2\") "
Nov 04 01:02:34.626398 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.625971    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca" pod "11d80979-9d84-49d8-9f7b-efde82ebb7b2" (UID: "11d80979-9d84-49d8-9f7b-efde82ebb7b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:02:34.695671 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.695602    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5" pod "11d80979-9d84-49d8-9f7b-efde82ebb7b2" (UID: "11d80979-9d84-49d8-9f7b-efde82ebb7b2"). InnerVolumeSpecName "kube-api-access-hfmw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:02:34.696043 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.695991    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca" pod "11d80979-9d84-49d8-9f7b-efde82ebb7b2" (UID: "11d80979-9d84-49d8-9f7b-efde82ebb7b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:02:34.710480 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.710393    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmw5\" (UniqueName: \"kubernetes.io/projected/11d80979-9d84-49d8-9f7b-efde82ebb7b2-kube-api-access-hfmw5\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:34.710480 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.710429    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:34.710480 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.710447    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d80979-9d84-49d8-9f7b-efde82ebb7b2-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:34.893673 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.893556    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m" event={"ID":"11d80979-9d84-49d8-9f7b-efde82ebb7b2","Type":"ContainerDied","Data":"05e491a3c2a2d9f0f46bb873574fd4d6f499d620b7e0924d47d7f49c13d7b830"}
Nov 04 01:02:34.893673 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.893623    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e491a3c2a2d9f0f46bb873574fd4d6f499d620b7e0924d47d7f49c13d7b830"
Nov 04 01:02:34.894007 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:34.893692    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"
Nov 04 01:02:41.279163 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.279124    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"]
Nov 04 01:02:41.279593 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.279579    5074 cpu_manager.go:414] "containerMap: removing container" podUID="11d80979-9d84-49d8-9f7b-efde82ebb7b2" containerName="revoke-keys"
Nov 04 01:02:41.279645 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.279594    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d80979-9d84-49d8-9f7b-efde82ebb7b2" containerName="revoke-keys"
Nov 04 01:02:41.279683 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.279661    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d80979-9d84-49d8-9f7b-efde82ebb7b2" containerName="revoke-keys"
Nov 04 01:02:41.283544 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.283517    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.291637 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.290241    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:02:41.291637 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.290348    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:02:41.292225 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.292093    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"]
Nov 04 01:02:41.427329 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.427289    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.427485 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.427361    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ktt\" (UniqueName: \"kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.427546 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.427475    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.529056 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.529017    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ktt\" (UniqueName: \"kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.529265 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.529179    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.529265 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.529257    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.529999 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.529977    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.531626 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.531602    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.547682 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.547656    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ktt\" (UniqueName: \"kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt\") pod \"maas-api-revoke-sub-e703a51b-ntd27\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.600794 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.600761    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:41.878016 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.877987    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"]
Nov 04 01:02:41.879539 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:02:41.879501    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda341cd2a_cca5_49f9_9e15_b5563ffb2dfc.slice/crio-ce6867016d328bff7659b7499a9ef2b884fedcf43e1818c20dd51e933af45715 WatchSource:0}: Error finding container ce6867016d328bff7659b7499a9ef2b884fedcf43e1818c20dd51e933af45715: Status 404 returned error can't find the container with id ce6867016d328bff7659b7499a9ef2b884fedcf43e1818c20dd51e933af45715
Nov 04 01:02:41.933190 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:41.933158    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27" event={"ID":"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc","Type":"ContainerStarted","Data":"ce6867016d328bff7659b7499a9ef2b884fedcf43e1818c20dd51e933af45715"}
Nov 04 01:02:42.939937 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:42.939900    5074 generic.go:356] "Generic (PLEG): container finished" podID="a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" containerID="adf25e449bb3958ee2e1c662a7f591f07af91edddf200150fda6fe9fa68cf34d" exitCode=0
Nov 04 01:02:42.940358 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:42.939947    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27" event={"ID":"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc","Type":"ContainerDied","Data":"adf25e449bb3958ee2e1c662a7f591f07af91edddf200150fda6fe9fa68cf34d"}
Nov 04 01:02:44.086684 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.086658    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:44.254646 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.254559    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca\") pod \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") "
Nov 04 01:02:44.254646 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.254604    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt\" (UniqueName: \"kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt\") pod \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") "
Nov 04 01:02:44.254909 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.254702    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca\") pod \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\" (UID: \"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc\") "
Nov 04 01:02:44.255100 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.255074    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca" pod "a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" (UID: "a341cd2a-cca5-49f9-9e15-b5563ffb2dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:02:44.256884 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.256857    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca" pod "a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" (UID: "a341cd2a-cca5-49f9-9e15-b5563ffb2dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:02:44.257037 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.256928    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt" pod "a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" (UID: "a341cd2a-cca5-49f9-9e15-b5563ffb2dfc"). InnerVolumeSpecName "kube-api-access-29ktt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:02:44.355694 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.355657    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:44.355694 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.355689    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:44.355694 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.355699    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ktt\" (UniqueName: \"kubernetes.io/projected/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc-kube-api-access-29ktt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:44.953057 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.953025    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"
Nov 04 01:02:44.953239 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.953064    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27" event={"ID":"a341cd2a-cca5-49f9-9e15-b5563ffb2dfc","Type":"ContainerDied","Data":"ce6867016d328bff7659b7499a9ef2b884fedcf43e1818c20dd51e933af45715"}
Nov 04 01:02:44.953239 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:44.953138    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce6867016d328bff7659b7499a9ef2b884fedcf43e1818c20dd51e933af45715"
Nov 04 01:02:51.516619 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.516566    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"]
Nov 04 01:02:51.517331 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.517297    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" containerName="revoke-keys"
Nov 04 01:02:51.517331 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.517323    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" containerName="revoke-keys"
Nov 04 01:02:51.517493 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.517408    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" containerName="revoke-keys"
Nov 04 01:02:51.520716 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.520699    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.526678 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.526629    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:02:51.526835 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.526805    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:02:51.528259 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.528234    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"]
Nov 04 01:02:51.625028 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.624988    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgkp\" (UniqueName: \"kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.625217 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.625105    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.625266 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.625215    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.726059 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.726023    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.726276 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.726107    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.726276 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.726153    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgkp\" (UniqueName: \"kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.726882 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.726858    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.728717 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.728687    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.739781 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.739724    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgkp\" (UniqueName: \"kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp\") pod \"maas-api-revoke-sub-82a7f0e1-w4vk7\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:51.836906 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:51.836818    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:52.114314 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:52.114284    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"]
Nov 04 01:02:52.116281 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:02:52.116245    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda142c740_e51e_4432_981c_efa15017b6c2.slice/crio-066e4ab835d3a170f17a05e23006af4650dbdca4396cc716a0b8815b26fb2580 WatchSource:0}: Error finding container 066e4ab835d3a170f17a05e23006af4650dbdca4396cc716a0b8815b26fb2580: Status 404 returned error can't find the container with id 066e4ab835d3a170f17a05e23006af4650dbdca4396cc716a0b8815b26fb2580
Nov 04 01:02:53.002803 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:53.002759    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7" event={"ID":"a142c740-e51e-4432-981c-efa15017b6c2","Type":"ContainerStarted","Data":"44929fbfa434677d80566ea47b402ad5abc58958b22fe091588d4dc708f6b5a0"}
Nov 04 01:02:53.003250 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:53.002834    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7" event={"ID":"a142c740-e51e-4432-981c-efa15017b6c2","Type":"ContainerStarted","Data":"066e4ab835d3a170f17a05e23006af4650dbdca4396cc716a0b8815b26fb2580"}
Nov 04 01:02:53.026908 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:53.026848    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7" podStartSLOduration=2.026830224 podStartE2EDuration="2.026830224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:53.02228222 +0000 UTC m=+5638.121882365" watchObservedRunningTime="2026-11-04 01:02:53.026830224 +0000 UTC m=+5638.126430369"
Nov 04 01:02:54.054123 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:54.054084    5074 generic.go:356] "Generic (PLEG): container finished" podID="a142c740-e51e-4432-981c-efa15017b6c2" containerID="44929fbfa434677d80566ea47b402ad5abc58958b22fe091588d4dc708f6b5a0" exitCode=0
Nov 04 01:02:54.054808 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:54.054780    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7" event={"ID":"a142c740-e51e-4432-981c-efa15017b6c2","Type":"ContainerDied","Data":"44929fbfa434677d80566ea47b402ad5abc58958b22fe091588d4dc708f6b5a0"}
Nov 04 01:02:55.363463 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.363433    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:55.383056 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.383024    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp\" (UniqueName: \"kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp\") pod \"a142c740-e51e-4432-981c-efa15017b6c2\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") "
Nov 04 01:02:55.383205 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.383076    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca\") pod \"a142c740-e51e-4432-981c-efa15017b6c2\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") "
Nov 04 01:02:55.383205 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.383129    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca\") pod \"a142c740-e51e-4432-981c-efa15017b6c2\" (UID: \"a142c740-e51e-4432-981c-efa15017b6c2\") "
Nov 04 01:02:55.384052 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.383592    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca" pod "a142c740-e51e-4432-981c-efa15017b6c2" (UID: "a142c740-e51e-4432-981c-efa15017b6c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:02:55.385983 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.385953    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp" pod "a142c740-e51e-4432-981c-efa15017b6c2" (UID: "a142c740-e51e-4432-981c-efa15017b6c2"). InnerVolumeSpecName "kube-api-access-kjgkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:02:55.386319 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.386291    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca" pod "a142c740-e51e-4432-981c-efa15017b6c2" (UID: "a142c740-e51e-4432-981c-efa15017b6c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:02:55.484121 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.484085    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:55.484121 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.484117    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjgkp\" (UniqueName: \"kubernetes.io/projected/a142c740-e51e-4432-981c-efa15017b6c2-kube-api-access-kjgkp\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:55.484121 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:55.484127    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a142c740-e51e-4432-981c-efa15017b6c2-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:02:56.077732 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:56.077704    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"
Nov 04 01:02:56.077921 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:56.077764    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7" event={"ID":"a142c740-e51e-4432-981c-efa15017b6c2","Type":"ContainerDied","Data":"066e4ab835d3a170f17a05e23006af4650dbdca4396cc716a0b8815b26fb2580"}
Nov 04 01:02:56.077921 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:02:56.077810    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066e4ab835d3a170f17a05e23006af4650dbdca4396cc716a0b8815b26fb2580"
Nov 04 01:02:58.021185 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:02:58.021149    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:03:01.752047 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.752010    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"]
Nov 04 01:03:01.752479 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.752456    5074 cpu_manager.go:414] "containerMap: removing container" podUID="a142c740-e51e-4432-981c-efa15017b6c2" containerName="revoke-keys"
Nov 04 01:03:01.752479 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.752471    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a142c740-e51e-4432-981c-efa15017b6c2" containerName="revoke-keys"
Nov 04 01:03:01.752569 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.752537    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a142c740-e51e-4432-981c-efa15017b6c2" containerName="revoke-keys"
Nov 04 01:03:01.756451 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.756424    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.762552 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.762524    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:03:01.762797 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.762610    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:03:01.764521 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.764494    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"]
Nov 04 01:03:01.841760 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.841713    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.842040 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.841767    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.842040 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.841839    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55knk\" (UniqueName: \"kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.942893 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.942857    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.943094 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.942904    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.943094 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.942944    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55knk\" (UniqueName: \"kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.943611 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.943582    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.945301 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.945275    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:01.956337 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:01.956312    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55knk\" (UniqueName: \"kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk\") pod \"maas-api-revoke-sub-94711568-72tzk\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:02.075576 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:02.075490    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:02.352649 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:02.352613    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"]
Nov 04 01:03:02.354846 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:03:02.354811    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9437ecaf_d2ee_47c3_b5e6_00519624360e.slice/crio-744f0977f42b6a382321ee101906d755194a7e69d8a145c65faca67864ac68aa WatchSource:0}: Error finding container 744f0977f42b6a382321ee101906d755194a7e69d8a145c65faca67864ac68aa: Status 404 returned error can't find the container with id 744f0977f42b6a382321ee101906d755194a7e69d8a145c65faca67864ac68aa
Nov 04 01:03:02.591472 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:02.591335    5074 scope.go:122] "RemoveContainer" containerID="cd965a491a70d4f2b113dfe47c0e780934c1f75c7de3f75e6ff4abe070488a4f"
Nov 04 01:03:03.141425 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:03.141377    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk" event={"ID":"9437ecaf-d2ee-47c3-b5e6-00519624360e","Type":"ContainerStarted","Data":"6436c92341ee949a68bf415b0accd2acfd00f93964a984e8da7702d5c44a1875"}
Nov 04 01:03:03.143151 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:03.143122    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk" event={"ID":"9437ecaf-d2ee-47c3-b5e6-00519624360e","Type":"ContainerStarted","Data":"744f0977f42b6a382321ee101906d755194a7e69d8a145c65faca67864ac68aa"}
Nov 04 01:03:03.172876 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:03.172812    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk" podStartSLOduration=2.172794562 podStartE2EDuration="2.172794562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:03.169138211 +0000 UTC m=+5648.268738380" watchObservedRunningTime="2026-11-04 01:03:03.172794562 +0000 UTC m=+5648.272394707"
Nov 04 01:03:04.111767 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:04.110863    5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kchxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:03:04.111767 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:04.110973    5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kchxb" podUID="c332db91-1798-471f-87d8-55217be24115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:03:04.160762 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:04.160083    5074 generic.go:356] "Generic (PLEG): container finished" podID="9437ecaf-d2ee-47c3-b5e6-00519624360e" containerID="6436c92341ee949a68bf415b0accd2acfd00f93964a984e8da7702d5c44a1875" exitCode=0
Nov 04 01:03:04.160762 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:04.160265    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk" event={"ID":"9437ecaf-d2ee-47c3-b5e6-00519624360e","Type":"ContainerDied","Data":"6436c92341ee949a68bf415b0accd2acfd00f93964a984e8da7702d5c44a1875"}
Nov 04 01:03:05.319044 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.319012    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:05.402510 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.402479    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk\" (UniqueName: \"kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk\") pod \"9437ecaf-d2ee-47c3-b5e6-00519624360e\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") "
Nov 04 01:03:05.402714 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.402529    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca\") pod \"9437ecaf-d2ee-47c3-b5e6-00519624360e\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") "
Nov 04 01:03:05.402714 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.402606    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca\") pod \"9437ecaf-d2ee-47c3-b5e6-00519624360e\" (UID: \"9437ecaf-d2ee-47c3-b5e6-00519624360e\") "
Nov 04 01:03:05.403017 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.402990    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca" pod "9437ecaf-d2ee-47c3-b5e6-00519624360e" (UID: "9437ecaf-d2ee-47c3-b5e6-00519624360e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:05.404713 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.404686    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk" pod "9437ecaf-d2ee-47c3-b5e6-00519624360e" (UID: "9437ecaf-d2ee-47c3-b5e6-00519624360e"). InnerVolumeSpecName "kube-api-access-55knk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:05.404852 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.404778    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca" pod "9437ecaf-d2ee-47c3-b5e6-00519624360e" (UID: "9437ecaf-d2ee-47c3-b5e6-00519624360e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:05.503965 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.503921    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55knk\" (UniqueName: \"kubernetes.io/projected/9437ecaf-d2ee-47c3-b5e6-00519624360e-kube-api-access-55knk\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:05.503965 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.503956    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:05.504231 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:05.503974    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9437ecaf-d2ee-47c3-b5e6-00519624360e-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:06.172187 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:06.172153    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk" event={"ID":"9437ecaf-d2ee-47c3-b5e6-00519624360e","Type":"ContainerDied","Data":"744f0977f42b6a382321ee101906d755194a7e69d8a145c65faca67864ac68aa"}
Nov 04 01:03:06.172187 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:06.172172    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94711568-72tzk"
Nov 04 01:03:06.172411 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:06.172204    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744f0977f42b6a382321ee101906d755194a7e69d8a145c65faca67864ac68aa"
Nov 04 01:03:11.983335 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.983295    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"]
Nov 04 01:03:11.983960 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.983938    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9437ecaf-d2ee-47c3-b5e6-00519624360e" containerName="revoke-keys"
Nov 04 01:03:11.983960 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.983962    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9437ecaf-d2ee-47c3-b5e6-00519624360e" containerName="revoke-keys"
Nov 04 01:03:11.984093 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.984048    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9437ecaf-d2ee-47c3-b5e6-00519624360e" containerName="revoke-keys"
Nov 04 01:03:11.987208 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.987159    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.000792 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.993809    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:03:12.000792 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.997059    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:03:12.000792 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:11.998377    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"]
Nov 04 01:03:12.070483 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.070447    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.070483 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.070486    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.070713 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.070513    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kxk2\" (UniqueName: \"kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.171902 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.171858    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.171902 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.171904    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.172177 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.172021    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kxk2\" (UniqueName: \"kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.172540 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.172517    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.174402 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.174375    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.185421 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.185397    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kxk2\" (UniqueName: \"kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2\") pod \"maas-api-revoke-sub-91870d2a-jnwfz\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.303695 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.303601    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:12.581628 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:12.581593    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"]
Nov 04 01:03:12.583052 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:03:12.583028    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9af6ae6_7c56_4002_a62a_5b3a1e6f204d.slice/crio-9e1f7ca0dc055039653ee627b33e1c15d1a6dfe2c4024b40335cdcfb70632179 WatchSource:0}: Error finding container 9e1f7ca0dc055039653ee627b33e1c15d1a6dfe2c4024b40335cdcfb70632179: Status 404 returned error can't find the container with id 9e1f7ca0dc055039653ee627b33e1c15d1a6dfe2c4024b40335cdcfb70632179
Nov 04 01:03:13.217468 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:13.217422    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz" event={"ID":"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d","Type":"ContainerStarted","Data":"9421dbbb2ed29c447eaeaeb3e7e730950b56d28b4bceae04fd66316d49af5c50"}
Nov 04 01:03:13.217934 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:13.217498    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz" event={"ID":"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d","Type":"ContainerStarted","Data":"9e1f7ca0dc055039653ee627b33e1c15d1a6dfe2c4024b40335cdcfb70632179"}
Nov 04 01:03:13.242935 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:13.242864    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz" podStartSLOduration=2.242843218 podStartE2EDuration="2.242843218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:13.236766134 +0000 UTC m=+5658.336366276" watchObservedRunningTime="2026-11-04 01:03:13.242843218 +0000 UTC m=+5658.342443364"
Nov 04 01:03:14.243764 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:14.239989    5074 generic.go:356] "Generic (PLEG): container finished" podID="d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" containerID="9421dbbb2ed29c447eaeaeb3e7e730950b56d28b4bceae04fd66316d49af5c50" exitCode=0
Nov 04 01:03:14.243764 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:14.240151    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz" event={"ID":"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d","Type":"ContainerDied","Data":"9421dbbb2ed29c447eaeaeb3e7e730950b56d28b4bceae04fd66316d49af5c50"}
Nov 04 01:03:15.638318 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.637996    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:15.775552 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.775504    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca\") pod \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") "
Nov 04 01:03:15.775783 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.775574    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca\") pod \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") "
Nov 04 01:03:15.775783 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.775669    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2\" (UniqueName: \"kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2\") pod \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\" (UID: \"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d\") "
Nov 04 01:03:15.777016 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.776989    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca" pod "d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" (UID: "d9af6ae6-7c56-4002-a62a-5b3a1e6f204d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:15.778827 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.778804    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca" pod "d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" (UID: "d9af6ae6-7c56-4002-a62a-5b3a1e6f204d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:15.778938 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.778861    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2" pod "d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" (UID: "d9af6ae6-7c56-4002-a62a-5b3a1e6f204d"). InnerVolumeSpecName "kube-api-access-8kxk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:15.877214 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.877138    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kxk2\" (UniqueName: \"kubernetes.io/projected/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-kube-api-access-8kxk2\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:15.877214 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.877167    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:15.877214 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:15.877178    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af6ae6-7c56-4002-a62a-5b3a1e6f204d-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:16.276412 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:16.276380    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz"
Nov 04 01:03:16.276637 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:16.276369    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91870d2a-jnwfz" event={"ID":"d9af6ae6-7c56-4002-a62a-5b3a1e6f204d","Type":"ContainerDied","Data":"9e1f7ca0dc055039653ee627b33e1c15d1a6dfe2c4024b40335cdcfb70632179"}
Nov 04 01:03:16.276701 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:16.276633    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e1f7ca0dc055039653ee627b33e1c15d1a6dfe2c4024b40335cdcfb70632179"
Nov 04 01:03:22.222338 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.222308    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"]
Nov 04 01:03:22.222801 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.222785    5074 cpu_manager.go:414] "containerMap: removing container" podUID="d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" containerName="revoke-keys"
Nov 04 01:03:22.222861 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.222803    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" containerName="revoke-keys"
Nov 04 01:03:22.222921 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.222912    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9af6ae6-7c56-4002-a62a-5b3a1e6f204d" containerName="revoke-keys"
Nov 04 01:03:22.226340 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.226320    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.228431 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.228395    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.228563 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.228441    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9xf\" (UniqueName: \"kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.228563 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.228532    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.231207 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.231186    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:03:22.233137 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.233111    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:03:22.237726 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.237705    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"]
Nov 04 01:03:22.329348 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.329309    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.329348 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.329349    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9xf\" (UniqueName: \"kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.329611 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.329384    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.330160 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.330134    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.331917 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.331890    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.342178 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.342155    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9xf\" (UniqueName: \"kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf\") pod \"maas-api-revoke-sub-be6088c7-rdvg8\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.543904 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.543797    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:22.828405 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:22.828370    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"]
Nov 04 01:03:22.829998 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:03:22.829959    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea961ad_fc9b_4209_9bc6_13948305db43.slice/crio-0880dfd8ba2381de582cf010f6d765a39e592b076674dbdb926423b148fbebb8 WatchSource:0}: Error finding container 0880dfd8ba2381de582cf010f6d765a39e592b076674dbdb926423b148fbebb8: Status 404 returned error can't find the container with id 0880dfd8ba2381de582cf010f6d765a39e592b076674dbdb926423b148fbebb8
Nov 04 01:03:23.321785 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:23.321731    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8" event={"ID":"9ea961ad-fc9b-4209-9bc6-13948305db43","Type":"ContainerStarted","Data":"2f92dd959783f2997e0bb17db92f5408585f024f2966b9342aa017ccf9ed725f"}
Nov 04 01:03:23.322189 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:23.321803    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8" event={"ID":"9ea961ad-fc9b-4209-9bc6-13948305db43","Type":"ContainerStarted","Data":"0880dfd8ba2381de582cf010f6d765a39e592b076674dbdb926423b148fbebb8"}
Nov 04 01:03:24.332623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:24.331943    5074 generic.go:356] "Generic (PLEG): container finished" podID="9ea961ad-fc9b-4209-9bc6-13948305db43" containerID="2f92dd959783f2997e0bb17db92f5408585f024f2966b9342aa017ccf9ed725f" exitCode=0
Nov 04 01:03:24.332623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:24.332063    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8" event={"ID":"9ea961ad-fc9b-4209-9bc6-13948305db43","Type":"ContainerDied","Data":"2f92dd959783f2997e0bb17db92f5408585f024f2966b9342aa017ccf9ed725f"}
Nov 04 01:03:25.781187 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.781157    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:25.883839 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.883793    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca\") pod \"9ea961ad-fc9b-4209-9bc6-13948305db43\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") "
Nov 04 01:03:25.884049 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.883887    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf\" (UniqueName: \"kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf\") pod \"9ea961ad-fc9b-4209-9bc6-13948305db43\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") "
Nov 04 01:03:25.884049 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.883946    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca\") pod \"9ea961ad-fc9b-4209-9bc6-13948305db43\" (UID: \"9ea961ad-fc9b-4209-9bc6-13948305db43\") "
Nov 04 01:03:25.884226 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.884197    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca" pod "9ea961ad-fc9b-4209-9bc6-13948305db43" (UID: "9ea961ad-fc9b-4209-9bc6-13948305db43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:25.884346 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.884325    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:25.886766 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.886687    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf" pod "9ea961ad-fc9b-4209-9bc6-13948305db43" (UID: "9ea961ad-fc9b-4209-9bc6-13948305db43"). InnerVolumeSpecName "kube-api-access-bc9xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:25.887355 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.887326    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca" pod "9ea961ad-fc9b-4209-9bc6-13948305db43" (UID: "9ea961ad-fc9b-4209-9bc6-13948305db43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:25.984914 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.984880    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc9xf\" (UniqueName: \"kubernetes.io/projected/9ea961ad-fc9b-4209-9bc6-13948305db43-kube-api-access-bc9xf\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:25.984914 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:25.984912    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea961ad-fc9b-4209-9bc6-13948305db43-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:26.379406 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:26.379323    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8"
Nov 04 01:03:26.379562 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:26.379328    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6088c7-rdvg8" event={"ID":"9ea961ad-fc9b-4209-9bc6-13948305db43","Type":"ContainerDied","Data":"0880dfd8ba2381de582cf010f6d765a39e592b076674dbdb926423b148fbebb8"}
Nov 04 01:03:26.379562 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:26.379454    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0880dfd8ba2381de582cf010f6d765a39e592b076674dbdb926423b148fbebb8"
Nov 04 01:03:28.020515 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:03:28.020480    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:03:32.462670 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.462632    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"]
Nov 04 01:03:32.463159 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.463142    5074 cpu_manager.go:414] "containerMap: removing container" podUID="9ea961ad-fc9b-4209-9bc6-13948305db43" containerName="revoke-keys"
Nov 04 01:03:32.463210 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.463160    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea961ad-fc9b-4209-9bc6-13948305db43" containerName="revoke-keys"
Nov 04 01:03:32.463251 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.463234    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea961ad-fc9b-4209-9bc6-13948305db43" containerName="revoke-keys"
Nov 04 01:03:32.466753 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.466722    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.473716 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.473690    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:03:32.473925 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.473703    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:03:32.477402 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.477367    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"]
Nov 04 01:03:32.544914 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.544881    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8lt\" (UniqueName: \"kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.545093 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.544922    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.545093 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.545048    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.645763 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.645699    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.645983 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.645861    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8lt\" (UniqueName: \"kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.645983 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.645888    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.646585 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.646558    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.648223 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.648194    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.659146 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.659116    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8lt\" (UniqueName: \"kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt\") pod \"maas-api-revoke-sub-675c3671-pv6ld\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:32.787568 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:32.787473    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:33.065876 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:33.065850    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"]
Nov 04 01:03:33.067326 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:03:33.067294    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d530b5_96ea_48a1_ad7d_bc9698cb3ee7.slice/crio-deafa5820af5e5e6c6c3672e5e5893d8c3a09ea7a7c7a30d224e045ea193cfbb WatchSource:0}: Error finding container deafa5820af5e5e6c6c3672e5e5893d8c3a09ea7a7c7a30d224e045ea193cfbb: Status 404 returned error can't find the container with id deafa5820af5e5e6c6c3672e5e5893d8c3a09ea7a7c7a30d224e045ea193cfbb
Nov 04 01:03:33.420869 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:33.420768    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld" event={"ID":"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7","Type":"ContainerStarted","Data":"1b6c932fc90deaa2cb11c11c350cc193a63d3521f5edc048e70b12bb6a0af17d"}
Nov 04 01:03:33.420869 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:33.420823    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld" event={"ID":"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7","Type":"ContainerStarted","Data":"deafa5820af5e5e6c6c3672e5e5893d8c3a09ea7a7c7a30d224e045ea193cfbb"}
Nov 04 01:03:33.442710 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:33.442618    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld" podStartSLOduration=1.442605499 podStartE2EDuration="1.442605499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:33.440012872 +0000 UTC m=+5678.539613017" watchObservedRunningTime="2026-11-04 01:03:33.442605499 +0000 UTC m=+5678.542205642"
Nov 04 01:03:34.432564 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:34.432522    5074 generic.go:356] "Generic (PLEG): container finished" podID="86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" containerID="1b6c932fc90deaa2cb11c11c350cc193a63d3521f5edc048e70b12bb6a0af17d" exitCode=0
Nov 04 01:03:34.433129 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:34.432616    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld" event={"ID":"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7","Type":"ContainerDied","Data":"1b6c932fc90deaa2cb11c11c350cc193a63d3521f5edc048e70b12bb6a0af17d"}
Nov 04 01:03:35.859460 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.859134    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:35.889726 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.889696    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca\") pod \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") "
Nov 04 01:03:35.889938 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.889770    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca\") pod \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") "
Nov 04 01:03:35.889938 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.889858    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt\" (UniqueName: \"kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt\") pod \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\" (UID: \"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7\") "
Nov 04 01:03:35.890224 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.890197    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca" pod "86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" (UID: "86d530b5-96ea-48a1-ad7d-bc9698cb3ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:35.892177 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.892152    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt" pod "86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" (UID: "86d530b5-96ea-48a1-ad7d-bc9698cb3ee7"). InnerVolumeSpecName "kube-api-access-hq8lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:35.892290 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.892189    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca" pod "86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" (UID: "86d530b5-96ea-48a1-ad7d-bc9698cb3ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:35.991089 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.991054    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8lt\" (UniqueName: \"kubernetes.io/projected/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-kube-api-access-hq8lt\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:35.991089 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.991089    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:35.991300 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:35.991103    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d530b5-96ea-48a1-ad7d-bc9698cb3ee7-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:36.475823 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:36.475786    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld" event={"ID":"86d530b5-96ea-48a1-ad7d-bc9698cb3ee7","Type":"ContainerDied","Data":"deafa5820af5e5e6c6c3672e5e5893d8c3a09ea7a7c7a30d224e045ea193cfbb"}
Nov 04 01:03:36.475823 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:36.475810    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675c3671-pv6ld"
Nov 04 01:03:36.476053 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:36.475851    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deafa5820af5e5e6c6c3672e5e5893d8c3a09ea7a7c7a30d224e045ea193cfbb"
Nov 04 01:03:42.709798 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.709762    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"]
Nov 04 01:03:42.710373 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.710353    5074 cpu_manager.go:414] "containerMap: removing container" podUID="86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" containerName="revoke-keys"
Nov 04 01:03:42.710420 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.710375    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" containerName="revoke-keys"
Nov 04 01:03:42.710483 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.710473    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d530b5-96ea-48a1-ad7d-bc9698cb3ee7" containerName="revoke-keys"
Nov 04 01:03:42.713694 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.713667    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.720270 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.719994    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:03:42.720535 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.720501    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:03:42.724009 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.723985    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"]
Nov 04 01:03:42.761994 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.761954    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.762157 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.762103    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9xr\" (UniqueName: \"kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.762225 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.762166    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.863114 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.863071    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.863324 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.863167    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9xr\" (UniqueName: \"kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.863324 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.863191    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.863878 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.863857    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.865647 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.865616    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:42.876466 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:42.876441    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9xr\" (UniqueName: \"kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr\") pod \"maas-api-revoke-sub-d1c8fad3-k97m4\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:43.031310 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:43.031228    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:43.320965 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:43.320935    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"]
Nov 04 01:03:43.323035 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:03:43.323008    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40957dd_9e36_4fb9_9dc2_e8844fee2637.slice/crio-7023cc9157ccb20e5d1b21deb6b7c7771e98419e4d109ecb01cbc1f72cc5b8ac WatchSource:0}: Error finding container 7023cc9157ccb20e5d1b21deb6b7c7771e98419e4d109ecb01cbc1f72cc5b8ac: Status 404 returned error can't find the container with id 7023cc9157ccb20e5d1b21deb6b7c7771e98419e4d109ecb01cbc1f72cc5b8ac
Nov 04 01:03:43.519055 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:43.519014    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4" event={"ID":"c40957dd-9e36-4fb9-9dc2-e8844fee2637","Type":"ContainerStarted","Data":"7023cc9157ccb20e5d1b21deb6b7c7771e98419e4d109ecb01cbc1f72cc5b8ac"}
Nov 04 01:03:44.529038 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:44.528941    5074 generic.go:356] "Generic (PLEG): container finished" podID="c40957dd-9e36-4fb9-9dc2-e8844fee2637" containerID="fbe4041ceb5afcf5b3b979b88031d3cbb00ad404c2d6c4bbeef49d3fd8b7377b" exitCode=0
Nov 04 01:03:44.529571 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:44.529117    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4" event={"ID":"c40957dd-9e36-4fb9-9dc2-e8844fee2637","Type":"ContainerDied","Data":"fbe4041ceb5afcf5b3b979b88031d3cbb00ad404c2d6c4bbeef49d3fd8b7377b"}
Nov 04 01:03:45.940761 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:45.916683    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:45.996198 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:45.996161    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca\") pod \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") "
Nov 04 01:03:45.996453 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:45.996225    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca\") pod \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") "
Nov 04 01:03:45.996453 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:45.996293    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr\" (UniqueName: \"kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr\") pod \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\" (UID: \"c40957dd-9e36-4fb9-9dc2-e8844fee2637\") "
Nov 04 01:03:45.996623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:45.996592    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca" pod "c40957dd-9e36-4fb9-9dc2-e8844fee2637" (UID: "c40957dd-9e36-4fb9-9dc2-e8844fee2637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:46.072762 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.052871    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca" pod "c40957dd-9e36-4fb9-9dc2-e8844fee2637" (UID: "c40957dd-9e36-4fb9-9dc2-e8844fee2637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:46.072762 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.053319    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr" pod "c40957dd-9e36-4fb9-9dc2-e8844fee2637" (UID: "c40957dd-9e36-4fb9-9dc2-e8844fee2637"). InnerVolumeSpecName "kube-api-access-5d9xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:46.097583 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.097553    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:46.097583 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.097581    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40957dd-9e36-4fb9-9dc2-e8844fee2637-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:46.097780 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.097593    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9xr\" (UniqueName: \"kubernetes.io/projected/c40957dd-9e36-4fb9-9dc2-e8844fee2637-kube-api-access-5d9xr\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:46.552883 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.552838    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4" event={"ID":"c40957dd-9e36-4fb9-9dc2-e8844fee2637","Type":"ContainerDied","Data":"7023cc9157ccb20e5d1b21deb6b7c7771e98419e4d109ecb01cbc1f72cc5b8ac"}
Nov 04 01:03:46.553105 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.552903    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7023cc9157ccb20e5d1b21deb6b7c7771e98419e4d109ecb01cbc1f72cc5b8ac"
Nov 04 01:03:46.553105 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:46.552976    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c8fad3-k97m4"
Nov 04 01:03:52.953717 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.953680    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"]
Nov 04 01:03:52.954163 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.954147    5074 cpu_manager.go:414] "containerMap: removing container" podUID="c40957dd-9e36-4fb9-9dc2-e8844fee2637" containerName="revoke-keys"
Nov 04 01:03:52.954214 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.954164    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40957dd-9e36-4fb9-9dc2-e8844fee2637" containerName="revoke-keys"
Nov 04 01:03:52.954263 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.954253    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40957dd-9e36-4fb9-9dc2-e8844fee2637" containerName="revoke-keys"
Nov 04 01:03:52.957637 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.957618    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:52.964176 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.964154    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:03:52.964372 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.964347    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:03:52.967727 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:52.967703    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"]
Nov 04 01:03:53.081233 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.081187    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.081423 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.081275    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vs64\" (UniqueName: \"kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.081532 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.081417    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.182334 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.182294    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.182492 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.182384    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vs64\" (UniqueName: \"kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.182492 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.182430    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.183134 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.183114    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.184819 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.184796    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.195171 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.195151    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vs64\" (UniqueName: \"kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64\") pod \"maas-api-revoke-sub-66a76c5d-wcf9m\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.272877 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.272789    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:53.551233 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.551204    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"]
Nov 04 01:03:53.553507 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:03:53.553476    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fba9f22_162b_43c2_a2af_4d9e6a75cee7.slice/crio-be915ac8c824af04063241c452a25164b2aa975b11a3c1a6adafc061f332adb3 WatchSource:0}: Error finding container be915ac8c824af04063241c452a25164b2aa975b11a3c1a6adafc061f332adb3: Status 404 returned error can't find the container with id be915ac8c824af04063241c452a25164b2aa975b11a3c1a6adafc061f332adb3
Nov 04 01:03:53.592582 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:53.592556    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m" event={"ID":"1fba9f22-162b-43c2-a2af-4d9e6a75cee7","Type":"ContainerStarted","Data":"be915ac8c824af04063241c452a25164b2aa975b11a3c1a6adafc061f332adb3"}
Nov 04 01:03:54.604111 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:54.604064    5074 generic.go:356] "Generic (PLEG): container finished" podID="1fba9f22-162b-43c2-a2af-4d9e6a75cee7" containerID="07a9dd0ec762b7e092e873b5ba71c2df7dd85be5876b6a773cbc047ffd7c24fa" exitCode=0
Nov 04 01:03:54.604628 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:54.604271    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m" event={"ID":"1fba9f22-162b-43c2-a2af-4d9e6a75cee7","Type":"ContainerDied","Data":"07a9dd0ec762b7e092e873b5ba71c2df7dd85be5876b6a773cbc047ffd7c24fa"}
Nov 04 01:03:55.997655 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:55.997615    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:56.172320 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.172263    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca\") pod \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") "
Nov 04 01:03:56.172714 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.172681    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64\" (UniqueName: \"kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64\") pod \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") "
Nov 04 01:03:56.172929 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.172907    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca\") pod \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\" (UID: \"1fba9f22-162b-43c2-a2af-4d9e6a75cee7\") "
Nov 04 01:03:56.174072 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.174039    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca" pod "1fba9f22-162b-43c2-a2af-4d9e6a75cee7" (UID: "1fba9f22-162b-43c2-a2af-4d9e6a75cee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:56.176301 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.176239    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca" pod "1fba9f22-162b-43c2-a2af-4d9e6a75cee7" (UID: "1fba9f22-162b-43c2-a2af-4d9e6a75cee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:56.177580 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.177521    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64" pod "1fba9f22-162b-43c2-a2af-4d9e6a75cee7" (UID: "1fba9f22-162b-43c2-a2af-4d9e6a75cee7"). InnerVolumeSpecName "kube-api-access-9vs64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:56.274906 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.274679    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:56.274906 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.274777    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vs64\" (UniqueName: \"kubernetes.io/projected/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-kube-api-access-9vs64\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:56.274906 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.274802    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fba9f22-162b-43c2-a2af-4d9e6a75cee7-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:56.445766 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.445464    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 01:03:56.462512 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.462468    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 01:03:56.655223 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.655123    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m" event={"ID":"1fba9f22-162b-43c2-a2af-4d9e6a75cee7","Type":"ContainerDied","Data":"be915ac8c824af04063241c452a25164b2aa975b11a3c1a6adafc061f332adb3"}
Nov 04 01:03:56.655536 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.655514    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be915ac8c824af04063241c452a25164b2aa975b11a3c1a6adafc061f332adb3"
Nov 04 01:03:56.656508 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:03:56.655538    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66a76c5d-wcf9m"
Nov 04 01:03:58.020334 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:03:58.020298    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:04:03.195488 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.195445    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"]
Nov 04 01:04:03.196195 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.196173    5074 cpu_manager.go:414] "containerMap: removing container" podUID="1fba9f22-162b-43c2-a2af-4d9e6a75cee7" containerName="revoke-keys"
Nov 04 01:04:03.196306 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.196198    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fba9f22-162b-43c2-a2af-4d9e6a75cee7" containerName="revoke-keys"
Nov 04 01:04:03.196377 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.196315    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fba9f22-162b-43c2-a2af-4d9e6a75cee7" containerName="revoke-keys"
Nov 04 01:04:03.200303 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.200267    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.206687 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.206651    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"]
Nov 04 01:04:03.211269 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.211234    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:04:03.211495 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.211277    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:04:03.243660 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.243627    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpjl\" (UniqueName: \"kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.243896 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.243689    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.243896 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.243834    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.344407 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.344364    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.344623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.344484    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpjl\" (UniqueName: \"kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.344623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.344523    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.345277 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.345255    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.346926 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.346900    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.357763 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.357710    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpjl\" (UniqueName: \"kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl\") pod \"maas-api-revoke-sub-574ff67f-w5xfm\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.520704 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.520615    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:03.801682 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:03.801648    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"]
Nov 04 01:04:03.803341 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:04:03.803311    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261a0797_a14f_409c_b832_257953546b23.slice/crio-083a1d5742858e090f00fb5deb5742eca89138816df821ed477d41d8d2eb9d54 WatchSource:0}: Error finding container 083a1d5742858e090f00fb5deb5742eca89138816df821ed477d41d8d2eb9d54: Status 404 returned error can't find the container with id 083a1d5742858e090f00fb5deb5742eca89138816df821ed477d41d8d2eb9d54
Nov 04 01:04:04.708291 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:04.708246    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm" event={"ID":"261a0797-a14f-409c-b832-257953546b23","Type":"ContainerStarted","Data":"4f8d5e6e1082de59642c6477b0fc53636e5136d11d9739903707124eb6ced4b3"}
Nov 04 01:04:04.708959 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:04.708322    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm" event={"ID":"261a0797-a14f-409c-b832-257953546b23","Type":"ContainerStarted","Data":"083a1d5742858e090f00fb5deb5742eca89138816df821ed477d41d8d2eb9d54"}
Nov 04 01:04:05.753623 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:05.753583    5074 generic.go:356] "Generic (PLEG): container finished" podID="261a0797-a14f-409c-b832-257953546b23" containerID="4f8d5e6e1082de59642c6477b0fc53636e5136d11d9739903707124eb6ced4b3" exitCode=0
Nov 04 01:04:05.765631 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:05.755159    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm" event={"ID":"261a0797-a14f-409c-b832-257953546b23","Type":"ContainerDied","Data":"4f8d5e6e1082de59642c6477b0fc53636e5136d11d9739903707124eb6ced4b3"}
Nov 04 01:04:07.044004 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.043942    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:07.085952 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.085915    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl\" (UniqueName: \"kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl\") pod \"261a0797-a14f-409c-b832-257953546b23\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") "
Nov 04 01:04:07.086173 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.086032    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca\") pod \"261a0797-a14f-409c-b832-257953546b23\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") "
Nov 04 01:04:07.086173 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.086101    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca\") pod \"261a0797-a14f-409c-b832-257953546b23\" (UID: \"261a0797-a14f-409c-b832-257953546b23\") "
Nov 04 01:04:07.086634 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.086603    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca" pod "261a0797-a14f-409c-b832-257953546b23" (UID: "261a0797-a14f-409c-b832-257953546b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:04:07.088299 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.088268    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca" pod "261a0797-a14f-409c-b832-257953546b23" (UID: "261a0797-a14f-409c-b832-257953546b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:04:07.088299 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.088267    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl" pod "261a0797-a14f-409c-b832-257953546b23" (UID: "261a0797-a14f-409c-b832-257953546b23"). InnerVolumeSpecName "kube-api-access-wmpjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:04:07.187759 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.187697    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261a0797-a14f-409c-b832-257953546b23-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:07.187759 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.187733    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261a0797-a14f-409c-b832-257953546b23-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:07.187759 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.187769    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpjl\" (UniqueName: \"kubernetes.io/projected/261a0797-a14f-409c-b832-257953546b23-kube-api-access-wmpjl\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:07.783163 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.783129    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm"
Nov 04 01:04:07.783344 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.783129    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574ff67f-w5xfm" event={"ID":"261a0797-a14f-409c-b832-257953546b23","Type":"ContainerDied","Data":"083a1d5742858e090f00fb5deb5742eca89138816df821ed477d41d8d2eb9d54"}
Nov 04 01:04:07.783344 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:07.783258    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="083a1d5742858e090f00fb5deb5742eca89138816df821ed477d41d8d2eb9d54"
Nov 04 01:04:13.432650 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.432615    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"]
Nov 04 01:04:13.433138 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.433090    5074 cpu_manager.go:414] "containerMap: removing container" podUID="261a0797-a14f-409c-b832-257953546b23" containerName="revoke-keys"
Nov 04 01:04:13.433138 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.433102    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261a0797-a14f-409c-b832-257953546b23" containerName="revoke-keys"
Nov 04 01:04:13.433242 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.433168    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="261a0797-a14f-409c-b832-257953546b23" containerName="revoke-keys"
Nov 04 01:04:13.436343 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.436321    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.442111 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.442087    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:04:13.442375 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.442344    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 01:04:13.444201 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.444171    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"]
Nov 04 01:04:13.536585 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.536555    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.536800 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.536637    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgpx8\" (UniqueName: \"kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.536800 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.536689    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.637326 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.637291    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgpx8\" (UniqueName: \"kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.637512 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.637337    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.637512 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.637401    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.638222 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.638199    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.639890 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.639863    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.650911 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.650884    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgpx8\" (UniqueName: \"kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8\") pod \"maas-api-revoke-sub-71b8a6ab-2xzrd\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:13.755404 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:13.755364    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:14.037999 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:04:14.037958    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257d0a65_46bd_4fbe_9096_82593946f9d1.slice/crio-681360dea7fa5864ba4661f082b16b22788114e72e7d757b7c942d95e30d2c68 WatchSource:0}: Error finding container 681360dea7fa5864ba4661f082b16b22788114e72e7d757b7c942d95e30d2c68: Status 404 returned error can't find the container with id 681360dea7fa5864ba4661f082b16b22788114e72e7d757b7c942d95e30d2c68
Nov 04 01:04:14.038549 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:14.038400    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"]
Nov 04 01:04:14.912765 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:14.912375    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd" event={"ID":"257d0a65-46bd-4fbe-9096-82593946f9d1","Type":"ContainerStarted","Data":"e37fe19475a7d090528334d3f0bf6fe1cf568b78088b653a6dae63511ae6d41f"}
Nov 04 01:04:14.912765 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:14.912460    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd" event={"ID":"257d0a65-46bd-4fbe-9096-82593946f9d1","Type":"ContainerStarted","Data":"681360dea7fa5864ba4661f082b16b22788114e72e7d757b7c942d95e30d2c68"}
Nov 04 01:04:14.954453 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:14.954373    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd" podStartSLOduration=1.954331157 podStartE2EDuration="1.954331157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:14.934149275 +0000 UTC m=+5720.033749445" watchObservedRunningTime="2026-11-04 01:04:14.954331157 +0000 UTC m=+5720.053931302"
Nov 04 01:04:15.959763 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:15.959038    5074 generic.go:356] "Generic (PLEG): container finished" podID="257d0a65-46bd-4fbe-9096-82593946f9d1" containerID="e37fe19475a7d090528334d3f0bf6fe1cf568b78088b653a6dae63511ae6d41f" exitCode=0
Nov 04 01:04:15.959763 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:15.959249    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd" event={"ID":"257d0a65-46bd-4fbe-9096-82593946f9d1","Type":"ContainerDied","Data":"e37fe19475a7d090528334d3f0bf6fe1cf568b78088b653a6dae63511ae6d41f"}
Nov 04 01:04:17.107117 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.107089    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:17.112152 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.112126    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8\" (UniqueName: \"kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8\") pod \"257d0a65-46bd-4fbe-9096-82593946f9d1\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") "
Nov 04 01:04:17.112277 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.112216    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca\") pod \"257d0a65-46bd-4fbe-9096-82593946f9d1\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") "
Nov 04 01:04:17.112327 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.112289    5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca\") pod \"257d0a65-46bd-4fbe-9096-82593946f9d1\" (UID: \"257d0a65-46bd-4fbe-9096-82593946f9d1\") "
Nov 04 01:04:17.112643 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.112619    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca" pod "257d0a65-46bd-4fbe-9096-82593946f9d1" (UID: "257d0a65-46bd-4fbe-9096-82593946f9d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:04:17.114486 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.114454    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8" pod "257d0a65-46bd-4fbe-9096-82593946f9d1" (UID: "257d0a65-46bd-4fbe-9096-82593946f9d1"). InnerVolumeSpecName "kube-api-access-sgpx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:04:17.114594 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.114497    5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca" pod "257d0a65-46bd-4fbe-9096-82593946f9d1" (UID: "257d0a65-46bd-4fbe-9096-82593946f9d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:04:17.213853 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.213817    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-openshift-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:17.213853 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.213849    5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgpx8\" (UniqueName: \"kubernetes.io/projected/257d0a65-46bd-4fbe-9096-82593946f9d1-kube-api-access-sgpx8\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:17.213853 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.213860    5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d0a65-46bd-4fbe-9096-82593946f9d1-maas-api-xks-ca\") on node \"ip-10-0-138-239.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:17.971653 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.971623    5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd"
Nov 04 01:04:17.971851 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.971622    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b8a6ab-2xzrd" event={"ID":"257d0a65-46bd-4fbe-9096-82593946f9d1","Type":"ContainerDied","Data":"681360dea7fa5864ba4661f082b16b22788114e72e7d757b7c942d95e30d2c68"}
Nov 04 01:04:17.971851 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:17.971771    5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="681360dea7fa5864ba4661f082b16b22788114e72e7d757b7c942d95e30d2c68"
Nov 04 01:04:28.028939 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:04:28.028896    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:04:32.428386 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:32.428333    5074 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal" podUID="14f459d1824c1ea630aa90a99306c647" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 04 01:04:34.027859 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:34.027808    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"]
Nov 04 01:04:34.030620 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:34.030589    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237795a-8ksnf"]
Nov 04 01:04:35.016818 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:35.016778    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b198ee-1a28-4c23-92c6-11344e0f71db" path="/var/lib/kubelet/pods/43b198ee-1a28-4c23-92c6-11344e0f71db/volumes"
Nov 04 01:04:44.038459 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:44.038410    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"]
Nov 04 01:04:44.043245 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:44.043212    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4a30ad-zct5j"]
Nov 04 01:04:45.016274 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:04:45.016232    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c12640-f7c7-4371-9f20-7c51a9d522e2" path="/var/lib/kubelet/pods/84c12640-f7c7-4371-9f20-7c51a9d522e2/volumes"
Nov 04 01:04:58.021826 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:04:58.021785    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:05:01.972168 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:01.972132    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 01:05:02.167086 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:02.167052    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 01:05:02.379621 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:02.379536    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 01:05:02.663471 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:02.663376    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-wktkm_ac114d7f-26b3-43c7-b52e-c273efafdcb7/manager/0.log"
Nov 04 01:05:02.896615 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:02.896585    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 01:05:03.318242 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:03.318208    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 01:05:14.955336 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:14.955299    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1efc0ce8-gtsvx_b4274aba-0c64-481b-be04-5462b6924c60/revoke-keys/0.log"
Nov 04 01:05:15.150942 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:15.150909    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-36e9ad82-p2ngq_038a62eb-a9e0-426d-aee0-6a161bcf619c/revoke-keys/0.log"
Nov 04 01:05:15.339253 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:15.339163    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3b5b3ff2-24k6m_11d80979-9d84-49d8-9f7b-efde82ebb7b2/revoke-keys/0.log"
Nov 04 01:05:15.533293 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:15.533265    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-574ff67f-w5xfm_261a0797-a14f-409c-b832-257953546b23/revoke-keys/0.log"
Nov 04 01:05:15.726304 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:15.726270    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-66a76c5d-wcf9m_1fba9f22-162b-43c2-a2af-4d9e6a75cee7/revoke-keys/0.log"
Nov 04 01:05:15.904646 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:15.904616    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-675c3671-pv6ld_86d530b5-96ea-48a1-ad7d-bc9698cb3ee7/revoke-keys/0.log"
Nov 04 01:05:16.103122 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:16.103046    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-71b8a6ab-2xzrd_257d0a65-46bd-4fbe-9096-82593946f9d1/revoke-keys/0.log"
Nov 04 01:05:16.300575 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:16.300546    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-82a7f0e1-w4vk7_a142c740-e51e-4432-981c-efa15017b6c2/revoke-keys/0.log"
Nov 04 01:05:16.489843 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:16.489809    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8842b7b2-pzzg7_be8f8558-1e83-42ae-8538-ef7904dd7496/revoke-keys/0.log"
Nov 04 01:05:16.679340 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:16.679316    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-91870d2a-jnwfz_d9af6ae6-7c56-4002-a62a-5b3a1e6f204d/revoke-keys/0.log"
Nov 04 01:05:16.860283 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:16.860198    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94711568-72tzk_9437ecaf-d2ee-47c3-b5e6-00519624360e/revoke-keys/0.log"
Nov 04 01:05:17.053752 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:17.053718    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-be6088c7-rdvg8_9ea961ad-fc9b-4209-9bc6-13948305db43/revoke-keys/0.log"
Nov 04 01:05:17.385984 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:17.385952    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d1430007-rdtxf_0516671c-96d7-4255-81e1-4df6e378ba09/revoke-keys/0.log"
Nov 04 01:05:17.563422 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:17.563387    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d1c8fad3-k97m4_c40957dd-9e36-4fb9-9dc2-e8844fee2637/revoke-keys/0.log"
Nov 04 01:05:17.749734 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:17.749691    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e703a51b-ntd27_a341cd2a-cca5-49f9-9e15-b5563ffb2dfc/revoke-keys/0.log"
Nov 04 01:05:17.935500 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:17.935466    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eac89823-m9wg5_5b626d5a-6d3e-4edf-9448-9e6e77bf7784/revoke-keys/0.log"
Nov 04 01:05:18.136296 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:18.136208    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c2d85_a1194887-d1c7-4fd0-8d54-cb43a3f60790/postgres/0.log"
Nov 04 01:05:28.020969 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:05:28.020927    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:05:30.294724 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:30.294690    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qnvzl_9c62bf5c-48e7-4250-ba5d-5e1a29efbe26/manager/0.log"
Nov 04 01:05:40.887840 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:40.887733    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pmdm_91e01291-c87c-47db-a09d-400263ff937c/manager/0.log"
Nov 04 01:05:42.486900 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:42.486866    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485ggl2s_350ff29e-6ece-4954-b340-80ef23b349dc/istio-proxy/0.log"
Nov 04 01:05:43.840592 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:43.840554    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/1.log"
Nov 04 01:05:44.231360 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:44.231324    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-m6wnb_eda3869d-7ca3-4325-b816-dd5e5153ba6c/istio-proxy/0.log"
Nov 04 01:05:44.440285 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:44.440252    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nlkqs_060dc557-cd0d-437a-b84e-6e52bb4e79a1/istio-proxy/0.log"
Nov 04 01:05:45.231786 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:45.231751    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk_f702aa70-1ecd-4e89-a575-04034b778897/istio-proxy/0.log"
Nov 04 01:05:45.767627 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:45.767598    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb58b79c9-bksx8_698112a3-0157-43a7-9ea6-30e554d8be51/router/0.log"
Nov 04 01:05:46.426084 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:46.426050    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/0.log"
Nov 04 01:05:52.028090 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:52.028051    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"]
Nov 04 01:05:52.032311 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:52.032281    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac89823-m9wg5"]
Nov 04 01:05:53.016113 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:05:53.016080    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b626d5a-6d3e-4edf-9448-9e6e77bf7784" path="/var/lib/kubelet/pods/5b626d5a-6d3e-4edf-9448-9e6e77bf7784/volumes"
Nov 04 01:05:58.022574 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:05:58.022528    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:06:02.635508 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:02.635468    5074 scope.go:122] "RemoveContainer" containerID="d34276bbbba95aca9a878c6566fcdef74f4099389cac20ecb1ee0f8338192e91"
Nov 04 01:06:02.645236 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:02.645211    5074 scope.go:122] "RemoveContainer" containerID="fef697f24d645656d84b7717773c2631183c9e52f02fede8cd07ee6a2b6f725b"
Nov 04 01:06:03.029769 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:03.029726    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"]
Nov 04 01:06:03.032106 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:03.032083    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e9ad82-p2ngq"]
Nov 04 01:06:05.016229 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:05.016188    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038a62eb-a9e0-426d-aee0-6a161bcf619c" path="/var/lib/kubelet/pods/038a62eb-a9e0-426d-aee0-6a161bcf619c/volumes"
Nov 04 01:06:19.642998 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.642957    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-595p9/must-gather-4jhkl"]
Nov 04 01:06:19.643425 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.643413    5074 cpu_manager.go:414] "containerMap: removing container" podUID="257d0a65-46bd-4fbe-9096-82593946f9d1" containerName="revoke-keys"
Nov 04 01:06:19.643425 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.643424    5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257d0a65-46bd-4fbe-9096-82593946f9d1" containerName="revoke-keys"
Nov 04 01:06:19.643510 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.643506    5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="257d0a65-46bd-4fbe-9096-82593946f9d1" containerName="revoke-keys"
Nov 04 01:06:19.646958 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.646939    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.654891 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.654856    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/must-gather-4jhkl"]
Nov 04 01:06:19.666578 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.666552    5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-595p9\"/\"default-dockercfg-whg7p\""
Nov 04 01:06:19.672851 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.672825    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-595p9\"/\"openshift-service-ca.crt\""
Nov 04 01:06:19.672982 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.672853    5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-595p9\"/\"kube-root-ca.crt\""
Nov 04 01:06:19.724127 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.724088    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/18579090-f4a7-4dc0-a0d2-a389a135dbc5-must-gather-output\") pod \"must-gather-4jhkl\" (UID: \"18579090-f4a7-4dc0-a0d2-a389a135dbc5\") " pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.724319 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.724155    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsq2j\" (UniqueName: \"kubernetes.io/projected/18579090-f4a7-4dc0-a0d2-a389a135dbc5-kube-api-access-qsq2j\") pod \"must-gather-4jhkl\" (UID: \"18579090-f4a7-4dc0-a0d2-a389a135dbc5\") " pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.824991 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.824951    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/18579090-f4a7-4dc0-a0d2-a389a135dbc5-must-gather-output\") pod \"must-gather-4jhkl\" (UID: \"18579090-f4a7-4dc0-a0d2-a389a135dbc5\") " pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.825180 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.825032    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsq2j\" (UniqueName: \"kubernetes.io/projected/18579090-f4a7-4dc0-a0d2-a389a135dbc5-kube-api-access-qsq2j\") pod \"must-gather-4jhkl\" (UID: \"18579090-f4a7-4dc0-a0d2-a389a135dbc5\") " pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.825318 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.825295    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/18579090-f4a7-4dc0-a0d2-a389a135dbc5-must-gather-output\") pod \"must-gather-4jhkl\" (UID: \"18579090-f4a7-4dc0-a0d2-a389a135dbc5\") " pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.854278 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.854245    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsq2j\" (UniqueName: \"kubernetes.io/projected/18579090-f4a7-4dc0-a0d2-a389a135dbc5-kube-api-access-qsq2j\") pod \"must-gather-4jhkl\" (UID: \"18579090-f4a7-4dc0-a0d2-a389a135dbc5\") " pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:19.975925 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:19.975885    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/must-gather-4jhkl"
Nov 04 01:06:20.254887 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:20.254856    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/must-gather-4jhkl"]
Nov 04 01:06:20.256203 ip-10-0-138-239 kubenswrapper[5074]: W1104 01:06:20.256178    5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18579090_f4a7_4dc0_a0d2_a389a135dbc5.slice/crio-da835072812ce2156a7225e4f0c1a534abeca015317ec44fa4b4866d94f2f5c7 WatchSource:0}: Error finding container da835072812ce2156a7225e4f0c1a534abeca015317ec44fa4b4866d94f2f5c7: Status 404 returned error can't find the container with id da835072812ce2156a7225e4f0c1a534abeca015317ec44fa4b4866d94f2f5c7
Nov 04 01:06:20.258092 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:20.258074    5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 01:06:20.731650 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:20.731616    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/must-gather-4jhkl" event={"ID":"18579090-f4a7-4dc0-a0d2-a389a135dbc5","Type":"ContainerStarted","Data":"da835072812ce2156a7225e4f0c1a534abeca015317ec44fa4b4866d94f2f5c7"}
Nov 04 01:06:21.744205 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:21.744162    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/must-gather-4jhkl" event={"ID":"18579090-f4a7-4dc0-a0d2-a389a135dbc5","Type":"ContainerStarted","Data":"a78e912e91a6ba4cad1f9d5b8b56ebe01163f9909949629c507da31467d5b525"}
Nov 04 01:06:21.744205 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:21.744216    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/must-gather-4jhkl" event={"ID":"18579090-f4a7-4dc0-a0d2-a389a135dbc5","Type":"ContainerStarted","Data":"afe25c87d554a197d121a176a86e7140561058e4f2e59a4032e6b9b3e79f84f5"}
Nov 04 01:06:21.766621 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:21.766566    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-595p9/must-gather-4jhkl" podStartSLOduration=1.714986179 podStartE2EDuration="2.766552445s" totalImagesPullingTime="1.051566266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:21.762427377 +0000 UTC m=+5846.862027523" watchObservedRunningTime="2026-11-04 01:06:21.766552445 +0000 UTC m=+5846.866152589"
Nov 04 01:06:23.151096 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:23.151058    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pcvlc_3e73b381-beec-43fe-ba3d-460efc605f8b/global-pull-secret-syncer/0.log"
Nov 04 01:06:23.205458 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:23.205433    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6mxbn_bbbb34c1-9e43-4640-82ae-660e1e365930/connection-checker/0.log"
Nov 04 01:06:23.235045 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:23.235019    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-c6qhd_5032904c-5a18-4f28-be3b-a02375b14113/connection-checker/0.log"
Nov 04 01:06:23.264780 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:23.264732    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qjdfr_a1f0dd5d-1903-4b2a-830c-05e93d384fda/connection-checker/0.log"
Nov 04 01:06:23.432776 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:23.432684    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b9jnr_47845d6a-a330-49f4-be05-d3a8aeb677dc/konnectivity-agent/0.log"
Nov 04 01:06:23.592093 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:23.592065    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-239.us-east-2.compute.internal_14f459d1824c1ea630aa90a99306c647/haproxy/0.log"
Nov 04 01:06:27.557058 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:27.557026    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 01:06:27.630825 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:27.630792    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 01:06:28.029552 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:06:28.029514    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:06:28.528973 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:28.528928    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 01:06:28.613330 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:28.613301    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-wktkm_ac114d7f-26b3-43c7-b52e-c273efafdcb7/manager/0.log"
Nov 04 01:06:28.624092 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:28.624068    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 01:06:29.174504 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:29.174471    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 01:06:30.029248 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:30.029205    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"]
Nov 04 01:06:30.031541 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:30.031512    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1430007-rdtxf"]
Nov 04 01:06:30.076523 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:30.076484    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 01:06:30.153541 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:30.153507    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 01:06:31.018041 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:31.018002    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0516671c-96d7-4255-81e1-4df6e378ba09" path="/var/lib/kubelet/pods/0516671c-96d7-4255-81e1-4df6e378ba09/volumes"
Nov 04 01:06:31.069308 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:31.069277    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 01:06:31.211463 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:31.211425    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-wktkm_ac114d7f-26b3-43c7-b52e-c273efafdcb7/manager/0.log"
Nov 04 01:06:31.225313 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:31.225286    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 01:06:31.784775 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:31.784722    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 01:06:32.679665 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:32.679616    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 01:06:32.766314 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:32.766252    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 01:06:33.705440 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:33.705404    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 01:06:33.844650 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:33.844609    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-wktkm_ac114d7f-26b3-43c7-b52e-c273efafdcb7/manager/0.log"
Nov 04 01:06:33.858349 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:33.858322    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 01:06:34.424180 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:34.424138    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 01:06:35.421341 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:35.421303    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485ggl2s_350ff29e-6ece-4954-b340-80ef23b349dc/istio-proxy/0.log"
Nov 04 01:06:41.027769 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.027724    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"]
Nov 04 01:06:41.032933 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.032899    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8842b7b2-pzzg7"]
Nov 04 01:06:41.033944 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.033917    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/0.log"
Nov 04 01:06:41.040174 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.040148    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/1.log"
Nov 04 01:06:41.076712 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.076684    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-m6wnb_eda3869d-7ca3-4325-b816-dd5e5153ba6c/istio-proxy/0.log"
Nov 04 01:06:41.090495 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.090462    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nlkqs_060dc557-cd0d-437a-b84e-6e52bb4e79a1/istio-proxy/0.log"
Nov 04 01:06:41.154333 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.154305    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk_f702aa70-1ecd-4e89-a575-04034b778897/istio-proxy/0.log"
Nov 04 01:06:41.206991 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:41.206961    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb58b79c9-bksx8_698112a3-0157-43a7-9ea6-30e554d8be51/router/0.log"
Nov 04 01:06:43.017866 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:43.017827    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8f8558-1e83-42ae-8538-ef7904dd7496" path="/var/lib/kubelet/pods/be8f8558-1e83-42ae-8538-ef7904dd7496/volumes"
Nov 04 01:06:44.690760 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:44.690085    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qnvzl_9c62bf5c-48e7-4250-ba5d-5e1a29efbe26/manager/0.log"
Nov 04 01:06:44.829759 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:44.827043    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pmdm_91e01291-c87c-47db-a09d-400263ff937c/manager/0.log"
Nov 04 01:06:49.566880 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:49.566841    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qnvzl_9c62bf5c-48e7-4250-ba5d-5e1a29efbe26/manager/0.log"
Nov 04 01:06:49.694432 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:49.693674    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pmdm_91e01291-c87c-47db-a09d-400263ff937c/manager/0.log"
Nov 04 01:06:50.956169 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:50.956133    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ngdxl_4741d3bf-14a8-4118-a61e-32dd236755dd/cluster-monitoring-operator/0.log"
Nov 04 01:06:51.244703 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:51.244624    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-czggl_b78183df-7e18-4a6d-b121-b25605f9d0e7/node-exporter/0.log"
Nov 04 01:06:51.278830 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:51.278802    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-czggl_b78183df-7e18-4a6d-b121-b25605f9d0e7/kube-rbac-proxy/0.log"
Nov 04 01:06:51.308209 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:51.308168    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-czggl_b78183df-7e18-4a6d-b121-b25605f9d0e7/init-textfile/0.log"
Nov 04 01:06:51.840185 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:51.840148    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qtr79_1848d733-9923-43d8-8caa-11fb3d30cb24/prometheus-operator-admission-webhook/0.log"
Nov 04 01:06:53.181006 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:53.180971    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-cs8jk_dd79947d-8f7a-4a12-8e8b-c8623c2e0e7d/networking-console-plugin/0.log"
Nov 04 01:06:53.778378 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:53.778343    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/1.log"
Nov 04 01:06:53.783601 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:53.783579    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6sdkj_e4e0a5aa-fe33-44ab-b2b9-811ddffea0fb/console-operator/2.log"
Nov 04 01:06:54.299576 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:54.299545    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nsh2h_001ac09b-53c8-4437-bc59-d1785cc36723/download-server/0.log"
Nov 04 01:06:54.900043 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:54.900017    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-svxq9_59264120-38b0-4c6d-b3be-7d0340db0e34/volume-data-source-validator/0.log"
Nov 04 01:06:55.594086 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:55.594030    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-tbczr_8d7148dc-308e-4a93-8f1d-1f8c0172c327/manager/0.log"
Nov 04 01:06:55.606591 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:55.606564    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-tbczr_8d7148dc-308e-4a93-8f1d-1f8c0172c327/copy-manifests/0.log"
Nov 04 01:06:55.979069 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:55.979037    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4x94t_a32ab19d-6150-424c-8ce9-d3c8fcf7f967/dns/0.log"
Nov 04 01:06:56.007761 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:56.007707    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4x94t_a32ab19d-6150-424c-8ce9-d3c8fcf7f967/kube-rbac-proxy/0.log"
Nov 04 01:06:56.196280 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:56.196251    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c99qk_c4738a53-824f-4a8b-ab86-b160b9f14c0e/dns-node-resolver/0.log"
Nov 04 01:06:56.207280 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:56.207252    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 01:06:56.295819 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:56.295712    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 01:06:56.870055 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:56.870021    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29895840-b78h6_373f1058-e636-45d6-a53b-cdef5a009136/image-pruner/0.log"
Nov 04 01:06:56.952719 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:56.952673    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-795f84c5c9-ktnfg_5564e75b-0171-4100-ad93-6f6e6089f0e0/registry/0.log"
Nov 04 01:06:57.027378 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:57.027334    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdxlq_cd9bbc11-d972-4662-bb0d-b1d8744673c9/node-ca/0.log"
Nov 04 01:06:57.311708 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:57.311678    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 01:06:57.463112 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:57.463077    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-wktkm_ac114d7f-26b3-43c7-b52e-c273efafdcb7/manager/0.log"
Nov 04 01:06:57.478512 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:57.478484    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 01:06:58.031602 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:58.031554    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 01:06:58.033433 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:06:58.033407    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:06:58.111841 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:58.111804    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485ggl2s_350ff29e-6ece-4954-b340-80ef23b349dc/istio-proxy/0.log"
Nov 04 01:06:58.703156 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:06:58.703122    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485ggl2s_350ff29e-6ece-4954-b340-80ef23b349dc/istio-proxy/0.log"
Nov 04 01:07:02.671504 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:02.671468    5074 scope.go:122] "RemoveContainer" containerID="097776caa62412e54a330772aa345f0635d729855e4cdc5504107d9c673c52d6"
Nov 04 01:07:02.683896 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:02.683870    5074 scope.go:122] "RemoveContainer" containerID="9a69eca20e1e01fb25608e0fe820da98e5bf3fc9ca9b0b138beb4326a82cf55f"
Nov 04 01:07:06.331385 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.331297    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/0.log"
Nov 04 01:07:06.335796 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.335769    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/1.log"
Nov 04 01:07:06.406643 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.406610    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-m6wnb_eda3869d-7ca3-4325-b816-dd5e5153ba6c/istio-proxy/0.log"
Nov 04 01:07:06.441569 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.441520    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nlkqs_060dc557-cd0d-437a-b84e-6e52bb4e79a1/istio-proxy/0.log"
Nov 04 01:07:06.575358 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.575319    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk_f702aa70-1ecd-4e89-a575-04034b778897/istio-proxy/0.log"
Nov 04 01:07:06.583195 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.583134    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/0.log"
Nov 04 01:07:06.587607 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.587585    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-kchxb_c332db91-1798-471f-87d8-55217be24115/istio-proxy/1.log"
Nov 04 01:07:06.620249 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.620221    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-m6wnb_eda3869d-7ca3-4325-b816-dd5e5153ba6c/istio-proxy/0.log"
Nov 04 01:07:06.633807 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.633778    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nlkqs_060dc557-cd0d-437a-b84e-6e52bb4e79a1/istio-proxy/0.log"
Nov 04 01:07:06.683557 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.683520    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb58b79c9-bksx8_698112a3-0157-43a7-9ea6-30e554d8be51/router/0.log"
Nov 04 01:07:06.698118 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.698094    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-mh7hk_f702aa70-1ecd-4e89-a575-04034b778897/istio-proxy/0.log"
Nov 04 01:07:06.749062 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:06.749027    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb58b79c9-bksx8_698112a3-0157-43a7-9ea6-30e554d8be51/router/0.log"
Nov 04 01:07:08.961424 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:08.961390    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6mthf_3b107666-eaaa-4d08-839b-d3e91dd5868b/serve-healthcheck-canary/0.log"
Nov 04 01:07:09.553502 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:09.553468    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-7869h_69d478d4-7d64-4a68-823e-29c6b40b4a94/insights-operator/0.log"
Nov 04 01:07:09.555943 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:09.555919    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-7869h_69d478d4-7d64-4a68-823e-29c6b40b4a94/insights-operator/1.log"
Nov 04 01:07:09.767809 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:09.767774    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzv94_75649c60-b6dd-4fe0-96f7-ff34c763de46/kube-rbac-proxy/0.log"
Nov 04 01:07:09.796541 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:09.796514    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzv94_75649c60-b6dd-4fe0-96f7-ff34c763de46/exporter/0.log"
Nov 04 01:07:09.826250 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:09.826176    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzv94_75649c60-b6dd-4fe0-96f7-ff34c763de46/extractor/0.log"
Nov 04 01:07:09.868482 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:09.868448    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-w5bsk-9tpvz_96b959e5-0c65-457f-8f9b-d386854c3642/insights-gathering/0.log"
Nov 04 01:07:11.401878 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.401832    5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"]
Nov 04 01:07:11.409356 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.409328    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.415618 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.415589    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"]
Nov 04 01:07:11.569184 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.569145    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-proc\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.569184 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.569182    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-sys\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.569428 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.569206    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-lib-modules\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.569428 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.569365    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-podres\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.569428 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.569399    5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mnc\" (UniqueName: \"kubernetes.io/projected/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-kube-api-access-62mnc\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670323 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670229    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-proc\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670323 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670271    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-sys\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670323 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670293    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-lib-modules\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670640 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670350    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-podres\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670640 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670372    5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mnc\" (UniqueName: \"kubernetes.io/projected/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-kube-api-access-62mnc\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670640 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-proc\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670640 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670432    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-podres\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670640 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670376    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-sys\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.670640 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.670472    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-lib-modules\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.683587 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.683557    5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mnc\" (UniqueName: \"kubernetes.io/projected/4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e-kube-api-access-62mnc\") pod \"perf-node-gather-daemonset-s26pn\" (UID: \"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:11.726981 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:11.726940    5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:12.019609 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.019583    5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"]
Nov 04 01:07:12.170951 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.170908    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn" event={"ID":"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e","Type":"ContainerStarted","Data":"a57db967b3449b27815e50ca676b3e4ea083cdce79a13950ef132eccbc9f7868"}
Nov 04 01:07:12.171121 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.171017    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:12.171121 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.171045    5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn" event={"ID":"4ea4e3e3-81eb-4ec3-a590-1a5b0d95a34e","Type":"ContainerStarted","Data":"d60f657873b9579a1d379d3617e05f6d4d51cf8c1685aaf47abdf1c72dc9e50e"}
Nov 04 01:07:12.191459 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.191407    5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn" podStartSLOduration=1.191394834 podStartE2EDuration="1.191394834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:12.189179909 +0000 UTC m=+5897.288780065" watchObservedRunningTime="2026-11-04 01:07:12.191394834 +0000 UTC m=+5897.290994969"
Nov 04 01:07:12.374497 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.374402    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-tbczr_8d7148dc-308e-4a93-8f1d-1f8c0172c327/manager/0.log"
Nov 04 01:07:12.410828 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:12.410798    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-tbczr_8d7148dc-308e-4a93-8f1d-1f8c0172c327/copy-manifests/0.log"
Nov 04 01:07:14.310154 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:14.310125    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5s44m_5bc3460b-b5fc-4efd-816f-fcd07a49b000/manager/0.log"
Nov 04 01:07:14.411330 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:14.411296    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5hbhg_769a6ec6-ceed-444c-aba3-739a7430928d/manager/0.log"
Nov 04 01:07:15.456104 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:15.456065    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f5pqr_74f97340-55be-4356-80cd-123779b1de3c/manager/0.log"
Nov 04 01:07:15.568653 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:15.568615    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc8d5ff6-wktkm_ac114d7f-26b3-43c7-b52e-c273efafdcb7/manager/0.log"
Nov 04 01:07:15.594928 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:15.594901    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fvh4c_4a0f933a-4da9-40ef-b486-40abeda40321/server/0.log"
Nov 04 01:07:16.182545 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:16.182512    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b7bxn_bd35424f-d798-46c3-9897-3aa4059bc928/proxy/0.log"
Nov 04 01:07:18.190523 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:18.190485    5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-s26pn"
Nov 04 01:07:19.880819 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:19.880783    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qnvzl_9c62bf5c-48e7-4250-ba5d-5e1a29efbe26/manager/0.log"
Nov 04 01:07:19.951526 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:19.951494    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pmdm_91e01291-c87c-47db-a09d-400263ff937c/manager/0.log"
Nov 04 01:07:21.628821 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:21.628790    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-bs5gg_9763223d-ebce-4993-9932-d443787cd21c/migrator/0.log"
Nov 04 01:07:21.659894 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:21.659863    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nls48_c752b04e-c068-4c8c-94d3-405ff6ab27bb/migrator/0.log"
Nov 04 01:07:22.081407 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:22.081372    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj_13b0dc51-f5cc-4afd-821f-c3097be21e80/kube-storage-version-migrator-operator/0.log"
Nov 04 01:07:22.082705 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:22.082679    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-h5ssj_13b0dc51-f5cc-4afd-821f-c3097be21e80/kube-storage-version-migrator-operator/1.log"
Nov 04 01:07:23.296628 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.296599    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/kube-multus-additional-cni-plugins/0.log"
Nov 04 01:07:23.328644 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.328614    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/egress-router-binary-copy/0.log"
Nov 04 01:07:23.367350 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.367309    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/cni-plugins/0.log"
Nov 04 01:07:23.400035 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.400000    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/bond-cni-plugin/0.log"
Nov 04 01:07:23.445499 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.445461    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/routeoverride-cni/0.log"
Nov 04 01:07:23.481756 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.481704    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/whereabouts-cni-bincopy/0.log"
Nov 04 01:07:23.517066 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:23.517022    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqzg8_a3bbd0ed-0bf6-401c-8ea6-d759c3909adc/whereabouts-cni/0.log"
Nov 04 01:07:24.028841 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:24.028795    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"]
Nov 04 01:07:24.031503 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:24.031470    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efc0ce8-gtsvx"]
Nov 04 01:07:24.356934 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:24.356854    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jt2sr_a1e7a5aa-87e3-4afd-ae59-801a559622f6/kube-multus/0.log"
Nov 04 01:07:24.691680 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:24.691650    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6n2fd_1244c649-9fe7-40c5-9b55-443a2529c710/network-metrics-daemon/0.log"
Nov 04 01:07:24.714689 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:24.714664    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6n2fd_1244c649-9fe7-40c5-9b55-443a2529c710/kube-rbac-proxy/0.log"
Nov 04 01:07:25.017278 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:25.017207    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4274aba-0c64-481b-be04-5462b6924c60" path="/var/lib/kubelet/pods/b4274aba-0c64-481b-be04-5462b6924c60/volumes"
Nov 04 01:07:26.530476 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.530418    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/ovn-controller/0.log"
Nov 04 01:07:26.555510 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.555483    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/ovn-acl-logging/0.log"
Nov 04 01:07:26.586288 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.586262    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/kube-rbac-proxy-node/0.log"
Nov 04 01:07:26.615189 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.615171    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/kube-rbac-proxy-ovn-metrics/0.log"
Nov 04 01:07:26.644161 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.644139    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/northd/0.log"
Nov 04 01:07:26.672826 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.672804    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/nbdb/0.log"
Nov 04 01:07:26.702963 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.702942    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/sbdb/0.log"
Nov 04 01:07:26.919836 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:26.919770    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ncc9n_5bc8021d-2812-40f5-99ba-85e10f04ef4f/ovnkube-controller/0.log"
Nov 04 01:07:28.032051 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:07:28.032008    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"
Nov 04 01:07:28.830112 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:28.830082    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-xr79v_17673e8b-0dec-48d4-8843-65147e22372a/check-endpoints/0.log"
Nov 04 01:07:28.862591 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:28.862563    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7cjjg_d2d422d3-fe51-4985-8577-4ddfcc24ba8c/network-check-target-container/0.log"
Nov 04 01:07:30.309762 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:30.309681    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6czcz_fcd47e4a-ca1b-4557-82d0-200bdb384976/iptables-alerter/0.log"
Nov 04 01:07:31.309932 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:31.309902    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dxj77_3a0dae53-7734-414d-94ca-7bfd0bbc5d62/tuned/0.log"
Nov 04 01:07:33.677794 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:33.677734    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5s52h_40135659-f74d-4307-8bb3-574411040f5a/cluster-samples-operator/0.log"
Nov 04 01:07:33.700400 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:33.700372    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5s52h_40135659-f74d-4307-8bb3-574411040f5a/cluster-samples-operator-watch/0.log"
Nov 04 01:07:34.883459 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:34.883422    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-8smkc_3cb8c88a-1985-4ca8-a11f-7ab86cd87634/service-ca-operator/1.log"
Nov 04 01:07:34.884096 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:34.884077    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-8smkc_3cb8c88a-1985-4ca8-a11f-7ab86cd87634/service-ca-operator/0.log"
Nov 04 01:07:35.034556 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:35.034512    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"]
Nov 04 01:07:35.034785 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:35.034569    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5b3ff2-24k6m"]
Nov 04 01:07:35.325275 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:35.325244    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-d5bgc_ebc10a97-edbf-42d7-bfd5-caa1d573329e/service-ca-controller/0.log"
Nov 04 01:07:35.820253 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:35.820216    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-jbbn4_63593602-b03b-47c5-9d74-a95ff2abfddb/csi-driver/0.log"
Nov 04 01:07:35.849653 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:35.849625    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-jbbn4_63593602-b03b-47c5-9d74-a95ff2abfddb/csi-node-driver-registrar/0.log"
Nov 04 01:07:35.880242 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:35.880213    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-jbbn4_63593602-b03b-47c5-9d74-a95ff2abfddb/csi-liveness-probe/0.log"
Nov 04 01:07:37.021256 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:37.021218    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d80979-9d84-49d8-9f7b-efde82ebb7b2" path="/var/lib/kubelet/pods/11d80979-9d84-49d8-9f7b-efde82ebb7b2/volumes"
Nov 04 01:07:45.040090 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:45.040053    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"]
Nov 04 01:07:45.042262 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:45.042236    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703a51b-ntd27"]
Nov 04 01:07:47.016845 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:47.016809    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a341cd2a-cca5-49f9-9e15-b5563ffb2dfc" path="/var/lib/kubelet/pods/a341cd2a-cca5-49f9-9e15-b5563ffb2dfc/volumes"
Nov 04 01:07:50.566559 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:50.566525    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qnvzl_9c62bf5c-48e7-4250-ba5d-5e1a29efbe26/manager/0.log"
Nov 04 01:07:50.633839 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:50.633808    5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pmdm_91e01291-c87c-47db-a09d-400263ff937c/manager/0.log"
Nov 04 01:07:56.026980 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:56.026938    5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"]
Nov 04 01:07:56.029540 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:56.029518    5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a7f0e1-w4vk7"]
Nov 04 01:07:57.017402 ip-10-0-138-239 kubenswrapper[5074]: I1104 01:07:57.017364    5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a142c740-e51e-4432-981c-efa15017b6c2" path="/var/lib/kubelet/pods/a142c740-e51e-4432-981c-efa15017b6c2/volumes"
Nov 04 01:07:58.020386 ip-10-0-138-239 kubenswrapper[5074]: E1104 01:07:58.020348    5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dzv94" containerName="extractor"